Regression Trees

Regression trees are decision trees for numeric targets. Instead of predicting class labels at leaves, they predict a number. The leaf prediction is usually the average target value of training samples that reached that leaf.

Core difference from classification trees:

• Classification trees optimize impurity reduction (entropy/Gini).
• Regression trees optimize variance reduction (or equivalent MSE reduction).

Node split scoring: at each node, compute variance of the target at the parent node, then subtract weighted child variances after a candidate split:

variance reduction = Var(parent) - [w_left * Var(left) + w_right * Var(right)]

Choose the split with the highest reduction. This is the regression analogue of information gain.

Leaf prediction rule: once stopping criteria are met (depth limit, low variance reduction, low sample count), output the mean target of leaf samples. That mean is the model's prediction for every future sample routed to that leaf.

Production implication: regression trees are piecewise-constant models. They perform well when target behavior changes across interpretable segments but can be unstable with very small leaves or noisy labels.

Failure mode: over-deep trees can memorize small target fluctuations. Use depth/min-samples/min-gain controls and validation-based tuning.

Interview-Ready Deepening

Source-backed reinforcement: these points add detail beyond short-duration UI hints and emphasize production tradeoffs.

• Generalizing decision trees from class prediction to numeric prediction by minimizing weighted variance and predicting leaf averages.
• Regression trees are decision trees for numeric targets.
• Leaf prediction rule: once stopping criteria are met (depth limit, low variance reduction, low sample count), output the mean target of leaf samples.
• Regression trees optimize variance reduction (or equivalent MSE reduction).
• The leaf prediction is usually the average target value of training samples that reached that leaf.
• Node split scoring: at each node, compute variance of the target at the parent node, then subtract weighted child variances after a candidate split:
• That mean is the model's prediction for every future sample routed to that leaf.
• The decision tree is going to make a prediction based on taking the average of the weights in the training examples down here.

Tradeoffs You Should Be Able to Explain

• Higher recall often increases context noise; reranking and filtering are required to keep precision high.
• Smaller chunks improve semantic precision but can break cross-sentence context needed for accurate answers.
• Aggressive grounding reduces hallucinations but can increase abstentions when retrieval coverage is weak.

First-time learner note: Read each model as a dataflow system: inputs become representations, representations become scores, and scores become decisions through a chosen loss and thresholding policy.

Production note: Track three things relentlessly in ML systems: data shape contracts, evaluation methodology, and the operational meaning of the model's errors. Most expensive failures come from one of those three.