Module 10 — Hyperparameter tuning and an end-to-end project

Every model in this course carries dials: $\lambda$ for Ridge, $k$ for k-NN, max_depth for trees, learning_rate for boosting. These hyperparameters — set before training, not learned from data — often decide more than the choice of algorithm itself. This closing module shows how to tune them without cheating, then assembles the whole course into one complete project workflow.

Searching the hyperparameter space

Grid search: exhaustive but expensive

Grid search tries every combination from a declared grid, evaluating each by cross-validation:

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(
    pipe,                                  # the module 8 pipeline, always
    param_grid={"ridge__alpha": [0.01, 0.1, 1, 10, 100]},
    cv=5, scoring="neg_mean_squared_error",
)
grid.fit(X_train, y_train)
grid.best_params_, grid.best_score_

Its cost explodes combinatorially: 4 hyperparameters × 5 values = 625 combinations × 5 folds = 3,125 trainings. Fine for one or two hyperparameters, prohibitive beyond.

Random search: the smarter default

Random search draws combinations at random from distributions. Counter-intuitive but well established: at equal budget, it usually beats grid search when hyperparameters are numerous — because only a few of them really matter, and random draws explore each dimension's values far more densely than a grid does.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint

search = RandomizedSearchCV(
    pipe,
    param_distributions={
        "model__learning_rate": loguniform(0.01, 0.3),
        "model__max_depth": randint(3, 8),
        "model__subsample": [0.7, 0.8, 0.9],
    },
    n_iter=60, cv=5, scoring="roc_auc", n_jobs=-1, random_state=42,
)

Note loguniform for scale parameters — the difference between 0.01 and 0.1 matters more than between 0.2 and 0.3. Beyond these two, Bayesian optimization (Optuna) directs the search toward promising regions; worth it when each training is expensive.

Tuning without lying to yourself

Module 8's rules apply with full force. The search must run on the training data only, with the pipeline inside the cross-validation (preprocessing re-fitted per fold). And since the search score itself is optimistically biased — you kept the combination that flattered the validation folds — the final measurement happens once, on the sealed test set from module 1.

The end-to-end project: the whole course in order

Every piece has been covered; here is their assembly order — the checklist for any supervised project:

1. Frame (module 1): target, features available at prediction time, regression or classification, and the metric chosen up front (module 9);
2. Split: sealed test set, stratify if classification, temporal split if time data (module 8);
3. Baseline: a simple model — regularized linear or logistic regression (modules 2–3). It sets the bar and sometimes suffices;
4. Pipeline: preprocessing + model, leakage made impossible (module 8);
5. Climb in power: random forest as a robust reference (module 6), then boosting with early stopping to gain the last points (module 7);
6. Tune: random search with cross-validation, on the metric chosen in step 1;
7. Final measurement: one single pass on the test set — the reportable figure;
8. Explain: coefficients, permutation importance (module 6), sensible error analysis.

The baseline is not a formality

A tuned logistic regression sometimes lands within a point of tuned boosting — at a fraction of the complexity, with readable coefficients thrown in. Skipping the baseline means depriving yourself of the yardstick that tells you whether the extra complexity actually pays.

Summary

• Hyperparameters are set before training; they are tuned by searching, never by eye.
• Random search usually beats grid search at equal budget when dials are numerous; loguniform for scale parameters.
• The search runs on training data with the pipeline inside cross-validation; the test set is used once, at the very end.
• The workflow — frame, split, baseline, pipeline, escalate, tune, measure, explain — is the course's true takeaway.

One step remains: the recap and the 40-question exam validating the whole course.