Module 5 — Decision trees: reading and limits
The decision tree is the most human of models: a cascade of yes/no questions, exactly like a diagnostic guide. Anyone can read one — a rare asset. But this simplicity hides a structural flaw, overfitting, whose remedy (module 6: forests) has made trees the basic ingredient of today's best tabular models.
A cascade of questions that splits the space
A tree classifies by asking a series of questions on the features: "income > €30,000?", then depending on the answer, "age < 25?", and so on down to a leaf that gives the prediction. Each question splits the feature space in two; the leaves form regions where the model predicts the majority class (or the mean, in regression).
from sklearn.tree import DecisionTreeClassifier, plot_tree
model = DecisionTreeClassifier(max_depth=3).fit(X_train, y_train)
plot_tree(model, feature_names=features, filled=True) # the model, readable at a glance
Notable strength: no standardization needed (each question involves one feature at a time, no distances), and categorical or numerical features coexist without effort.
How the tree chooses its questions
At each node, the algorithm tries every possible split (each feature, each threshold) and keeps the one that produces the purest child groups — as homogeneous as possible in class terms. The standard purity measure is Gini impurity: it is 0 when a group contains only one class, maximal when classes are evenly mixed. At every stage the tree greedily picks the split that most reduces impurity.
This construction gives a readable by-product: summing each feature's contribution to impurity reduction yields the tree's feature importances (feature_importances_) — a first answer to "which features matter?".
The structural flaw: overfitting guaranteed
Left free, a tree keeps splitting until it isolates every training point in its own leaf: 100% accuracy on train, disastrous generalization. It's the archetype of the overfitting seen in the mathematics course — maximal variance. And that's not its only fragility: change a few rows of the dataset and the tree can pick a different first question, rebuilding the entire structure. A tree is unstable.
Containing the tree: pruning and constraints
You rein in a tree by bounding its growth:
| Parameter | Effect |
|---|---|
max_depth | limits the number of question levels |
min_samples_leaf | forbids near-empty leaves |
min_samples_split | requires enough points before splitting |
These constraints raise bias to lower variance — the usual trade-off, tuned by cross-validation.
Even pruned, a lone tree remains unstable and mediocre next to modern methods. Its real value: as an explanation tool (extracting readable business rules) and above all as the building block of ensembles. A hundred trees whose errors cancel out — that's the random forest, subject of the next module and a spectacular answer to the instability just diagnosed.
Summary
- A tree stacks yes/no questions and predicts in the leaves; readable at a glance, no standardization required.
- Splits are chosen by minimizing impurity (Gini); the by-product is the model's feature importances.
- Left free, a tree always overfits and stays unstable; it is contained with
max_depth,min_samples_leaf,min_samples_split. - Its true role: an explanation tool and the base ingredient of ensembles — forests and boosting to follow.
Next module: random forests, or how the vote of hundreds of imperfect trees produces a robust model.