Gradient Descent for Logistic Regression

Gradient descent for logistic regression keeps the same outer loop structure as linear regression, but uses logistic predictions.

Updates:

w_j := w_j - alpha*(1/m)*sum((ŷ_i-y_i)*x_ij)
b := b - alpha*(1/m)*sum(ŷ_i-y_i)

with ŷ_i = sigmoid(w⃗·x⃗_i + b).

Key point: same update shape, different prediction function and loss. This is why moving from linear to logistic code is mostly a model-head change plus BCE loss choice.

Production diagnostics:

• Monitor loss and calibration metrics (Brier/log loss) alongside accuracy.
• Use scaled features for faster convergence.
• Check class imbalance; consider class weights when positive class is rare.

Vectorized implementation: compute all logits in one matrix multiply, apply sigmoid, compute residual vector (ŷ-y), then backprop/update in batch.

Deepening Notes

Source-backed reinforcement: these points are extracted from the session source note to strengthen your theory intuition.

• You have now mastered: Linear regression Gradient descent Vectorization Feature scaling Feature engineering Polynomial regression Logistic regression Decision boundaries Log loss Gradient descent for classification You now understand core supervised learning.
• In fact, you'd be able to choose parameters that will result in the cost function being exactly equal to zero because the errors are zero on all five training examples.
• When I think about underfitting and overfitting, high bias and high variance.
• So far we've looked at underfitting and overfitting for linear regression model.
• Once again, this is an instance of overfitting and high variance because its model, despite doing very well on the training set, doesn't look like it'll generalize well to new examples.

Interview-Ready Deepening

Source-backed reinforcement: these points add detail beyond short-duration UI hints and emphasize production tradeoffs.

• Same update rule as linear regression — but with sigmoid applied underneath.
• Gradient descent for logistic regression keeps the same outer loop structure as linear regression, but uses logistic predictions.
• Linear regression: ŷ = w⃗·x⃗ + b, gradient = (ŷ−y)·x. Logistic regression: ŷ = σ(w⃗·x⃗ + b), gradient = (ŷ−y)·x. Same formula, different ŷ computation. The gradient descent loop is identical.
• Gradient Descent for Logistic Regression: Linear regression: ŷ = w⃗·x⃗ + b, gradient = (ŷ−y)·x. Logistic regression: ŷ = σ(w⃗·x⃗ + b), gradient = (ŷ−y)·x. Same formula, different ŷ computation. The gradient descent loop is identical.
• Updates: w_j := w_j - alpha*(1/m)*sum((ŷ_i-y_i)*x_ij) b := b - alpha*(1/m)*sum(ŷ_i-y_i) with ŷ_i = sigmoid(w⃗·x⃗_i + b) .
• This is why moving from linear to logistic code is mostly a model-head change plus BCE loss choice.
• Vectorized implementation: compute all logits in one matrix multiply, apply sigmoid, compute residual vector (ŷ-y), then backprop/update in batch.
• Key point: same update shape, different prediction function and loss.

Tradeoffs You Should Be Able to Explain

• More expressive models improve fit but can reduce interpretability and raise overfitting risk.
• Higher optimization speed can reduce training time but may increase instability if learning dynamics are not monitored.
• Feature-rich pipelines improve performance ceilings but increase maintenance and monitoring complexity.

First-time learner note: Read each model as a dataflow system: inputs become representations, representations become scores, and scores become decisions through a chosen loss and thresholding policy.

Production note: Track three things relentlessly in ML systems: data shape contracts, evaluation methodology, and the operational meaning of the model's errors. Most expensive failures come from one of those three.