Cost Function

The cost function (also called loss function) answers the question: 'How wrong is my model right now?'

It distils the model's performance on all training examples into a single number. The goal of training is to find the parameters (w, b) that make this number as small as possible.

Mean Squared Error (MSE) for linear regression:

J(w,b) = (1/2m) × Σ (ŷᵢ − yᵢ)²

Breaking this down step by step:

• ŷᵢ − yᵢ: the error for one training example (prediction minus true value)
• (ŷᵢ − yᵢ)²: squaring makes the error always positive and punishes large mistakes harder
• Σ ...: sum the squared errors for all m training examples
• (1/2m): average over m examples; the ½ is a calculus convenience that cancels the 2 from the derivative

Intuition: If the model predicts house prices perfectly for every example, J = 0. The worse the predictions, the higher J climbs. Training is the process of making J as close to 0 as possible.

Why square errors? Two reasons: (1) negative and positive errors don't cancel each other out, and (2) large errors get penalised much more heavily than small ones — a prediction that's off by 10 contributes 100 to the cost, while being off by 1 contributes only 1.

Deepening Notes

Source-backed reinforcement: these points are extracted from the session source note to strengthen your theory intuition.

• In order to implement linear regression the first key step is first to define something called a cost function.
• To introduce a little bit more terminology the w and b are called the parameters of the model.
• In machine learning parameters of the model are the variables you can adjust during training in order to improve the model.
• This difference is called the error, we're measuring how far off to prediction is from the target.
• This is also called the squared error cost function, and it's called this because you're taking the square of these error terms.

Interview-Ready Deepening

Source-backed reinforcement: these points add detail beyond short-duration UI hints and emphasize production tradeoffs.

• To build a cost function that doesn't automatically get bigger as the training set size gets larger by convention, we will compute the average squared error instead of the total squared error and we do that by dividing by m like this.
• Model predicts house prices [300K, 400K, 500K]. True prices are [280K, 420K, 480K]. MSE = average of [(20K)², (20K)², (20K)²] / 2 = single 'wrongness' score to minimise.
• By convention, the cost function that machine learning people use actually divides by 2 times m.
• In order to implement linear regression the first key step is first to define something called a cost function.
• This difference is called the error, we're measuring how far off to prediction is from the target.
• The cost function takes the prediction y hat and compares it to the target y by taking y hat minus y.
• This expression right here is the cost function and we're going to write J of wb to refer to the cost function.
• Intuition: If the model predicts house prices perfectly for every example, J = 0.

Tradeoffs You Should Be Able to Explain

• More expressive models improve fit but can reduce interpretability and raise overfitting risk.
• Higher optimization speed can reduce training time but may increase instability if learning dynamics are not monitored.
• Feature-rich pipelines improve performance ceilings but increase maintenance and monitoring complexity.

First-time learner note: Read each model as a dataflow system: inputs become representations, representations become scores, and scores become decisions through a chosen loss and thresholding policy.

Production note: Track three things relentlessly in ML systems: data shape contracts, evaluation methodology, and the operational meaning of the model's errors. Most expensive failures come from one of those three.