Regularisation — Math for Linear Regression

L2-regularized linear regression objective:

J(w,b) = (1/2m) * sum((ŷ_i - y_i)^2) + (lambda/2m) * sum(w_j^2)

Details that matter:

• bias term b is usually excluded from penalty.
• lambda term is normalized by m for scale consistency.
• regularization acts on weights, not labels/features.

Weight update with decay:

w_j := w_j*(1 - alpha*lambda/m) - alpha*(1/m)*sum((ŷ_i-y_i)*x_ij)

The first factor is weight decay. Each step slightly shrinks coefficient magnitude before fitting residual structure.

Practical insight: if features are not standardized, regularization acts unevenly because coefficient scales are not comparable. Standardize first, then tune lambda.

Operational check: monitor coefficient norms as lambda changes; exploding norms indicate weak regularization or unstable optimization settings.

Deepening Notes

Source-backed reinforcement: these points are extracted from the session source note to strengthen your theory intuition.

• You now understand: Overfitting Underfitting Bias vs variance Regularization intuition Mathematical regularization formulation You are now moving from “student level” to “engineer level”.
• In this video, we'll figure out how to get gradient descent to work with regularized linear regression.
• The first part is the usual squared error cost function, and now you have this additional regularization term, where Lambda is the regularization parameter, and you'd like to find parameters w and b that minimize the regularized cost function.
• Let's take these definitions for the derivatives and put them back into the expression on the left to write out the gradient descent algorithm for regularized linear regression.
• To implement gradient descent for regularized linear regression, this is what you would have your code do.

Interview-Ready Deepening

Source-backed reinforcement: these points add detail beyond short-duration UI hints and emphasize production tradeoffs.

• L2 penalty added to MSE; weight decay in the gradient update.
• With α=0.01, λ=1, m=100: decay factor = 1 − (0.01·1/100) = 1 − 0.0001 = 0.9999. Each step, w shrinks by 0.01% before the gradient update. Over 10,000 steps, this prevents w from growing unboundedly.
• This is the update for linear regression before we had regularization, and this is the term we saw in Week 2 of this course.
• In fact, the updates for a regularized linear regression look exactly the same, except that now the cost, J, is defined a bit differently.
• This is why this expression is used to compute the gradient in regularized linear regression.
• That's why the updated B remains the same as before, whereas the updated w changes because the regularization term causes us to try to shrink w_j.
• Recall that f of x for linear regression is defined as w dot x plus b or w dot product x plus b.
• Here is the update for w_j, for j equals 1 through n, and here's the update for b.

Tradeoffs You Should Be Able to Explain

• More expressive models improve fit but can reduce interpretability and raise overfitting risk.
• Higher optimization speed can reduce training time but may increase instability if learning dynamics are not monitored.
• Feature-rich pipelines improve performance ceilings but increase maintenance and monitoring complexity.

First-time learner note: Read each model as a dataflow system: inputs become representations, representations become scores, and scores become decisions through a chosen loss and thresholding policy.

Production note: Track three things relentlessly in ML systems: data shape contracts, evaluation methodology, and the operational meaning of the model's errors. Most expensive failures come from one of those three.