Learning Curves

Learning curves plot model error as a function of training-set size. Instead of asking only "how good is the model right now?", they ask "how is the model behaving as it gets more experience?" The horizontal axis is m_train, the number of training examples. The vertical axis is usually J_train and J_cv.

The surprising pattern: as the training set gets bigger, training error usually rises. That sounds wrong at first, but it makes sense: fitting one or two points perfectly is easy, while fitting hundreds of points perfectly is much harder. Cross-validation error usually falls because the model sees more representative data and generalizes better.

High-bias learning curve: both curves flatten early at a relatively high level. J_train is high, J_cv is a little higher, and the gap is small. This means the model is too simple. More data does not solve the problem because the model family itself cannot represent the pattern well enough.

High-variance learning curve: J_train stays low while J_cv is much higher, creating a large gap. As more data is added, the gap can shrink. This is the classic case where collecting more data is genuinely useful, because the model already has enough capacity and needs more examples to stop memorizing.

How to use this in practice: learning curves are a decision tool. If curves show high bias, do not launch a huge data-collection project first. Change model capacity, features, or regularization. If curves show high variance, more data, augmentation, or stronger regularization may pay off. The curve tells you which direction is worth your time.

Architecture note: most teams do not recompute full learning curves every day because training many sub-models is expensive. But the mental model is essential. A lot of strong ML decisions come from asking, "If I doubled data tomorrow, would the error really move?"

Interview-Ready Deepening

Source-backed reinforcement: these points add detail beyond short-duration UI hints and emphasize production tradeoffs.

• Cross-validation error usually falls because the model sees more representative data and generalizes better.
• In this example, just by getting more training data, allows the algorithm to go from relatively high cross-validation error to get much closer to human level performance.
• Architecture note: most teams do not recompute full learning curves every day because training many sub-models is expensive.
• Learning curves plot model error as a function of training-set size.
• This is the classic case where collecting more data is genuinely useful, because the model already has enough capacity and needs more examples to stop memorizing.
• Which is why as the training set gets bigger, the training error increases because it's harder to fit all of the training examples perfectly.
• That gives this conclusion, maybe a little bit surprising, that if a learning algorithm has high bias, getting more training data will not by itself hope that much.
• To summarize, if a learning algorithm suffers from high variance, then getting more training data is indeed likely to help.

Tradeoffs You Should Be Able to Explain

• More expressive models improve fit but can reduce interpretability and raise overfitting risk.
• Higher optimization speed can reduce training time but may increase instability if learning dynamics are not monitored.
• Feature-rich pipelines improve performance ceilings but increase maintenance and monitoring complexity.

First-time learner note: Read each model as a dataflow system: inputs become representations, representations become scores, and scores become decisions through a chosen loss and thresholding policy.

Production note: Track three things relentlessly in ML systems: data shape contracts, evaluation methodology, and the operational meaning of the model's errors. Most expensive failures come from one of those three.