Module 6 — Random forests and bootstrap aggregation

Module 5 left a diagnosis: the tree is readable but unstable and prone to overfitting. The remedy is one of the most beautiful ideas in machine learning: rather than seeking one perfect model, train hundreds of imperfect ones and have them vote. Individual errors, provided they are independent, cancel out in the aggregate. That is the random forest.

Bagging: the wisdom of crowds applied to models

Bagging (bootstrap aggregating) rests on two steps:

1. Bootstrap: draw, with replacement, several samples of the same size as the original dataset. Each resample sees certain rows repeated and others absent — so it is slightly different.
2. Aggregation: train one tree per resample, then combine — majority vote in classification, mean in regression.

Why does it work? Each deep tree has low bias but enormous variance. Averaging many trees whose errors differ divides the variance without raising the bias. Instability, the tree's flaw, becomes fuel: the more the trees differ from one another, the more effective the averaging.

The forest's extra: randomizing features too

Trees trained on bootstrap resamples remain too alike: if one feature is dominant, they all put it at the root and their errors correlate — averaging loses its power. The random forest adds a second level of randomness: at each node, only a random subset of features (typically $\sqrt{d}$) may be used for the split.

This forced decorrelation is the key to the method: trees explore different structures, their errors become more independent, and the aggregation gains full effect.

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(
    n_estimators=300,      # number of trees
    max_features="sqrt",   # features drawn at each node
    n_jobs=-1,             # parallel training
    random_state=42,
).fit(X_train, y_train)

An appreciable comfort: the forest works remarkably well with almost no tuning. n_estimators merely needs to be large enough (performance plateaus, never degrades, as trees are added — it only costs compute time).

Out-of-bag: free validation

Elegance of the bootstrap: each tree never saw about a third of the rows (those not drawn). These out-of-bag points serve as a personal test set for each tree, and aggregating gives an honest performance estimate without touching the test set or running cross-validation:

RandomForestClassifier(n_estimators=300, oob_score=True).fit(X_train, y_train).oob_score_

Feature importance, forest-scale

The forest aggregates its trees' importances (module 5) into a more stable measure of each feature's contribution — one of the most used tools for explaining a model. Caution though: importances spread out across correlated features, and impurity-based importance favors high-cardinality features. For a critical read, permutation importance (shuffling one feature's values and measuring the performance drop) is more reliable.

What the forest gives up

One tree was readable; three hundred are not. The forest trades module 5's transparency for robustness — importances and permutation partly compensate. Its other limit: prediction requires querying every tree, costly under real-time constraints. Boosting, in the next module, often does better still on tabular data.

Summary

• Bagging = bootstrap + aggregation: averaging many high-variance trees divides the variance without raising bias.
• The forest adds the random draw of features at each node, decorrelating the trees — the key to its effectiveness.
• The out-of-bag score provides honest validation for free, without cross-validation.
• Nearly tuning-free and robust, the forest loses the lone tree's readability; importances (ideally by permutation) compensate.

Next module: gradient boosting — building trees no longer in parallel but in sequence, each correcting the previous ones' errors.