#decision-trees — Supervised learning

A tree that carves the plane into rectangles: Gini, entropy, depth, pruning — and the overfitting you can see with your own eyes.

What you'll play with

Welcome to #decision-trees. On the left, the plane [-2, 2]²: 120 training points (blue and pink) and 60 validation points (smaller, translucent). On the right, a tree reduced to a root and two leaves: it has asked a single question — x ≤ 0.88 ? — and the plane is cut into two rectangles, one already almost pure, the other still mixed. This is twenty questions applied to a point cloud: pick a variable and a threshold, split, repeat on each half. Each leaf votes for its majority class, and the opacity of its rectangle tells you how pure it is.
Add a level: /split. Each still-impure leaf now asks its own question in turn — two chained questions, at most four rectangles.
One more level: /split. At depth 3, the tree could grow up to eight leaves, but it stops on its own wherever a leaf is already pure: nothing left to gain.
How does the tree choose its question? It tries every possible threshold on x and on y and keeps the one that most reduces the impurity of the two halves. Two measures exist: the Gini index (the default) and entropy. Compare: /criterion entropy.
Now let the tree grow as much as it wants: /depth 8. It will keep splitting until every leaf is pure — even if that means isolating a single point.
The classic fix: forbid leaves that are too small. /min-leaf 8 requires at least eight points per leaf — a form of preventive pruning, decided before splitting.
New terrain: /dataset checkerboard. Four alternating quadrants, a spread-out XOR: no straight line separates these two classes, logistic regression fails completely here.
Last strength: a tree can be read. Ask it a question: /predict 1 -1. The query point appears in yellow on the plane and its root → leaf path lights up in the tree, test after test.
Your turn: /prune to go back one level, /depth 2 to see the checkerboard solved in two questions, /noise 0.4 then /depth 8 for spectacular overfitting, /seed 7 for another draw (a greedy tree is unstable — that is what random forests exploit by averaging hundreds of trees), /dataset blobs to watch the staircase approximate a diagonal, /reset to start over. Next up: #knn, which decides without building any model, then #overfitting to meet the same trap with a neural network.

Channel commands

/depth <1..8> — Sets the maximum tree depth; the tree is retrained.
/split — Adds one level: maximum depth + 1.
/prune — Removes one level: maximum depth − 1.
/criterion <gini|entropy> — Picks the impurity measure: Gini index or entropy.
/min-leaf <1..20> — Minimum number of training points per leaf (pre-pruning).
/dataset <blobs|moons|checkerboard> — Switches the dataset (train and validation are regenerated).
/noise <0..0.5> — Point spread and fraction of flipped labels (datasets regenerated).
/seed <1..99> — Another random draw of the points (same distribution).
/predict <x=-2..2> <y=-2..2> — Places a query point and highlights its root → leaf path.
/reset — Returns to the initial state: moons, depth 1, Gini, no query.

Glossary

Decision tree: A model that classifies a point by asking it a chain of x ≤ threshold ? questions, from the root down to a leaf. Each question splits the space in two: the tree carves the plane into rectangles.
Node / leaf: An internal node holds a test (variable + threshold) and two children: yes on the left, no on the right. A leaf tests nothing more: it predicts the majority class of the training points it contains.
Gini impurity: Measures the class mixture at a node: 1 − Σ p². Equals 0 for a pure node and 0.5 for a 50/50 mix with two classes. It is the default criterion of CART and scikit-learn.
Entropy: Another mixture measure: −Σ p·log₂ p, in bits. Equals 0 for a pure node and 1 bit for a 50/50 mix. In practice, Gini and entropy produce very similar trees.
Information gain: Drop in impurity from a split: parent impurity minus the weighted average of the two children's impurity. At each node, the greedy algorithm tries every threshold and keeps the one with the largest gain.
Depth: Maximum number of questions between the root and a leaf. The deeper the tree, the finer it can split — and the more likely it is to memorise noise. It is the main hyperparameter of a tree.
Pruning: Simplifying a tree so it generalises better: either by preventing it from growing (maximum depth, minimum points per leaf — pre-pruning), or by removing branches after the fact that do not help validation (post-pruning).
Overfitting: When the model sticks to the training points so tightly that it memorises exceptions: training accuracy near 100%, validation flat or falling. On a tree it is visible: tiny rectangles around isolated points.
Random forest: An ensemble of hundreds of trees, each trained on a resampling of the data and a subset of the variables, whose votes are averaged. It fixes the main flaw of a single tree: its instability.
Interpretability: The ability to explain a prediction in human terms. A tree is interpretable by construction: the root → leaf path is a list of readable rules ("y ≤ −0.3 and x > 0.9 therefore pink").

Other channels in Supervised learning

#live-training — Six algorithms learning before your eyes, like a video: REC, timecode, subtitles, live metrics. Watching is free; touching the model is Premium.
#linear-regression — Fit a line: least squares, residuals, MSE, R² and gradient descent — the first brick of every supervised model.
#logistic-regression — Classify into two categories: sigmoid, decision boundary, threshold and log-loss — and why a line is not always enough.
#decision-trees — A tree that carves the plane into rectangles: Gini, entropy, depth, pruning — and the overfitting you can see with your own eyes.
#knn — k nearest neighbours: classify by resemblance, pick k, change the distance — and watch the boundary smooth out or shatter.
#svm-margins — Support vector machines: the widest possible margin, the C parameter, and the RBF kernel that curves the boundary.
#classification-metrics — Precision, recall, F1, confusion matrix, ROC and AUC: reading a classifier honestly, especially when classes are imbalanced.