Skip to main content

Module 6 — Automatic hyperparameter tuning

The scikit-learn script of module 5 has two knobs, n-estimators and max-depth. A grid over those two, with three values each, would launch nine jobs. On a real model with six or eight hyperparameters, that combinatorics collapses. SageMaker's Automatic Model Tuning solves the sizing problem by using a Bayesian search that spends compute where the search space looks promising.

The Tuner, on top of an existing estimator​

The tuning object wraps the estimator you already built. It does not change the training script; it repeatedly launches the estimator with different hyperparameter values, reads the metric SageMaker parsed from the standard output, and decides what to try next.

from sagemaker.tuner import HyperparameterTuner, IntegerParameter, ContinuousParameter, CategoricalParameter

hyperparameter_ranges = {
    "n-estimators": IntegerParameter(100, 500),
    "max-depth": IntegerParameter(3, 10),
    "learning-rate": ContinuousParameter(0.01, 0.3, scaling_type="Logarithmic"),
    "subsample": ContinuousParameter(0.5, 1.0),
    "loss": CategoricalParameter(["log_loss", "exponential"]),
}

tuner = HyperparameterTuner(
    estimator=sk,
    objective_metric_name="validation:auc",
    objective_type="Maximize",
    hyperparameter_ranges=hyperparameter_ranges,
    metric_definitions=[{"Name": "validation:auc", "Regex": r"validation-auc:\s*([0-9\.]+)"}],
    max_jobs=30,
    max_parallel_jobs=3,
    strategy="Bayesian",
    early_stopping_type="Auto",
)

tuner.fit({"train": train_input, "validation": val_input})

Every field carries a decision.

IntegerParameter, ContinuousParameter, CategoricalParameter map to the three shapes of a search space. For continuous parameters spanning several orders of magnitude — a learning rate from 0.001 to 0.3 — scaling_type="Logarithmic" samples uniformly in the log domain, which is what you want; a linear sampling would spend most of the compute at large values.

objective_metric_name must match a name declared in metric_definitions. A typo here produces a running tuner that reads nothing back and picks trials at random.

strategy="Bayesian" is the default and the best choice for expensive trials. SageMaker fits a Gaussian process to the metric surface and picks the next trial where the expected improvement is highest. Two alternatives exist: "Random" for a cheap baseline, and "Hyperband" for cases where a bad configuration reveals itself early — deep learning being the canonical example.

max_parallel_jobs=3 means three trials run at the same time. This is a genuine trade-off, not a free acceleration.

A serial Bayesian search updates its posterior after every completed trial, so trial N+1 uses everything learned by trial N. Running three trials in parallel means trials N+1, N+2 and N+3 are picked from the same posterior and cannot see each other's results. The search quality per trial drops with parallelism; the wall-clock speeds up.

The rule I use: max_parallel_jobs = 10 % of max_jobs, rounded up, capped at 10. With max_jobs=30, three parallel is a good balance. On a scarce GPU quota you may drop it to one, on a large budget where wall-clock matters more than quality you may push it to ten.

Early stopping​

early_stopping_type="Auto" kills trials whose learning curves look worse, at similar training progress, than the trials already completed. It cuts about a third of the compute on the churn problem with minimal impact on the best score. It requires the training script to report the objective metric at least twice, so SageMaker can compare progress; if the script prints the AUC only at the end, there is no curve to read and early stopping has no effect.

Warm start​

A common mistake is to run a tuning job, look at the best trial, and then run another tuning job on a slightly different range. The second job starts from scratch: it does not know the twenty-nine trials of the first one.

Warm start reuses those results:

from sagemaker.tuner import WarmStartConfig, WarmStartTypes

warm = WarmStartConfig(
    warm_start_type=WarmStartTypes.TRANSFER_LEARNING,
    parents={"churn-tuning-2026-09-06-12-34-56"},
)
tuner2 = HyperparameterTuner(..., warm_start_config=warm)

IDENTICAL_DATA_AND_ALGORITHM is faster and stricter — data must be the same. TRANSFER_LEARNING is more flexible — the new job may use different data or a modified script — and the prior is used only as a starting point.

Reading the results​

Tuning jobs write their results to a table you can query as a pandas DataFrame:

import sagemaker

analytics = sagemaker.HyperparameterTuningJobAnalytics(tuner.latest_tuning_job.name)
df = analytics.dataframe()

top = df.sort_values("FinalObjectiveValue", ascending=False).head(5)
print(top[["TrainingJobName", "FinalObjectiveValue", "n-estimators", "max-depth"]])

Three sanity checks on that table, in order:

Look at the objective distribution. If the best and the worst trials differ by 0.001 AUC, either your search space is too narrow to matter or your metric is dominated by noise; take the cheapest configuration and stop tuning.

Look at the boundary trials. A best trial whose n-estimators sits at the very top of the range (500) tells you the true optimum is outside the box you gave the tuner. Widen the range and warm-start.

Look at correlations. If max-depth and learning-rate are individually flat but a scatter plot reveals a diagonal ridge, the search found a compensation you had not seen. This is where tuning teaches you something.

A stopped tuner still incurs cost until the last trial finishes

Clicking "Stop" on a tuning job in the console prevents new trials from launching, but every trial already running keeps running to completion, and every instance already provisioned is billed until the job releases it. Set max_jobs conservatively and use early stopping rather than relying on the panic button.

Summary​

  • The Tuner wraps an estimator; it runs many training jobs and picks each configuration with a Bayesian model of the metric surface.
  • Ranges declare type, bounds and scaling; a learning rate ranges logarithmically, not linearly.
  • Parallelism trades quality for wall-clock; about ten percent of max_jobs is a reasonable default.
  • Early stopping cuts a third of the compute if the script reports the metric more than once; warm start avoids restarting from scratch across runs.

Next module: deploy the winning model as a real-time or serverless endpoint.