Measuring Purity: Entropy

To choose good splits, a tree needs a way to quantify how mixed or pure a node is. The source note uses entropy for this. Entropy is a scalar measure of impurity: low entropy means the node mostly contains one class, while high entropy means the node is mixed.

Binary-class intuition:

• If a node is all cats, entropy is 0.
• If a node is all dogs, entropy is 0.
• If a node is a 50-50 mix of cats and dogs, entropy is 1, which is the maximum impurity in the binary case.

Formal definition: let p1 be the fraction of positive examples in the node and p0 = 1 - p1 be the fraction of negatives. Then:

H(p1) = -p1 log2(p1) - p0 log2(p0)

Why the curve behaves this way: certainty produces low entropy, while uncertainty produces high entropy. A node that is almost all one class is easy to label. A node that is half one class and half the other is hard to label cleanly, so it has high impurity.

Examples from the source note: a node with 3 cats and 3 dogs has p1 = 0.5 and entropy 1. A node with 5 cats and 1 dog has lower entropy, around 0.65. A node with 6 cats and 0 dogs has entropy 0 because it is completely pure.

Implementation detail: when p1 = 0 or p0 = 0, the term 0 log(0) is treated as 0 by convention. This avoids numerical issues and gives the correct result that a pure node has zero entropy.

Why entropy matters operationally: the learning algorithm is trying to push training examples into cleaner and cleaner subsets. Entropy gives a numerical way to say whether a candidate split actually improved that cleanliness.

Architecture note: entropy is not the only impurity metric. Libraries may also use Gini impurity, which has a similar shape and similar purpose. What matters conceptually is not memorizing one formula; it is understanding that the tree needs a consistent impurity measure to compare candidate splits.

Failure mode: beginners often think "more branches means better tree." Not necessarily. A split is only useful if it produces child nodes that are meaningfully purer. Entropy helps you distinguish productive splitting from meaningless branching.

Interview-Ready Deepening

Source-backed reinforcement: these points add detail beyond short-duration UI hints and emphasize production tradeoffs.

• Entropy is the impurity measure that tells a decision tree how mixed a node is, with 0 meaning pure and 1 meaning maximally mixed in the binary case.
• Entropy is a scalar measure of impurity: low entropy means the node mostly contains one class, while high entropy means the node is mixed.
• If a node is a 50-50 mix of cats and dogs, entropy is 1, which is the maximum impurity in the binary case.
• This is actually quite impure and in particular this set is more impure than this set because it's closer to a 50-50 mix, which is why the impurity here is 0.92 as opposed to 0.65.
• To choose good splits, a tree needs a way to quantify how mixed or pure a node is.
• A node with 6 cats and 0 dogs has entropy 0 because it is completely pure.
• What matters conceptually is not memorizing one formula; it is understanding that the tree needs a consistent impurity measure to compare candidate splits.
• We're going to measure the impurity of a set of examples using a function called the entropy which looks like this.

Tradeoffs You Should Be Able to Explain

• More expressive models improve fit but can reduce interpretability and raise overfitting risk.
• Higher optimization speed can reduce training time but may increase instability if learning dynamics are not monitored.
• Feature-rich pipelines improve performance ceilings but increase maintenance and monitoring complexity.

First-time learner note: Read each model as a dataflow system: inputs become representations, representations become scores, and scores become decisions through a chosen loss and thresholding policy.

Production note: Track three things relentlessly in ML systems: data shape contracts, evaluation methodology, and the operational meaning of the model's errors. Most expensive failures come from one of those three.