Module 7 — Gradient boosting: XGBoost and LightGBM

The forest trains its trees in parallel and averages them. Boosting takes the opposite route: trees in sequence, each specializing in the errors of the previous ones. This obstinate accumulation of small corrections produces the most powerful models on tabular data — those that win competitions and equip industry.

The idea: a sequence of correctors

Boosting starts from a very simple prediction (the target's mean, say), then repeats:

1. Compute the current model's residual errors;
2. Train a small tree to predict those errors;
3. Add this corrective tree to the model, weighted by a learning rate;
4. Start again on the new errors.

Each tree is weak — shallow, barely better than chance — but the sequence relentlessly reduces the error. The name "gradient" is no accident: predicting the residuals amounts to following the gradient of the cost in model space; it is the gradient descent from the mathematics course, where each step is a tree.

Fundamental contrast with the forest: bagging reduces variance by averaging strong, independent trees; boosting reduces bias by accumulating weak, dependent trees. Consequence: boosting can overfit if allowed to correct too long — its guardrails are essential.

The learning rate, guardrail number one

The learning rate weights each tree's contribution:

$$F_{m}(x) = F_{m-1}(x) + \eta \cdot \text{tree}_m(x)$$

A small $\eta$ (0.05, say) forces small steps: more trees are needed, but generalization improves — hasty corrections overfit the noise. The classic pairing: low learning rate + many trees + early stopping.

XGBoost and LightGBM in practice

Two libraries dominate: XGBoost (the robust benchmark) and LightGBM (faster on large volumes, by growing trees leaf-first). Same logic, near-identical APIs:

import lightgbm as lgb
model = lgb.LGBMClassifier(
    n_estimators=2000,        # ceiling, early stopping will trim
    learning_rate=0.05,
    max_depth=4,              # weak trees
    subsample=0.8,            # row fraction per tree
    colsample_bytree=0.8,     # feature fraction per tree
)
model.fit(X_train, y_train,
          eval_set=[(X_val, y_val)],
          callbacks=[lgb.early_stopping(50)])

Early stopping watches the error on a validation set and halts adding trees as soon as it stops improving over 50 rounds: the number of trees is thereby tuned automatically. subsample and colsample_bytree inject the randomness that worked so well for the forest.

Hyperparameter	Role	Point of caution
learning_rate	correction-step size	small = better but slower
max_depth	tree complexity	3–6 usually suffices
n_estimators	number of trees	ceiling; delegate to early stopping
subsample / colsample	randomness	0.7–0.9, decorrelates the trees

Boosting or forest?

Forest: nearly tuning-free, robust, hard to derail — ideal baseline. Boosting: better peak performance on tabular data, at the cost of real tuning (module 10) and overfitting vigilance. Standard practice: forest first as reference, then boosting to gain the final points. In both cases the trees do the work; only the assembly philosophy changes.

Summary

• Boosting builds its trees in sequence, each correcting the previous ones' residuals — a gradient descent whose steps are trees.
• Bagging reduces variance, boosting reduces bias: powerful but can overfit without guardrails.
• The learning rate sets the step size: small + many trees + early stopping is the winning combo.
• XGBoost and LightGBM dominate tabular practice; max_depth, subsample, colsample complete the tuning.

Next module: cross-validation and data leakage — how to measure all these models' performance without fooling yourself.