Module 4 — Training jobs and built-in containers

The Parquet files from module 3 are on S3. This module trains the churn model on them using SageMaker's built-in XGBoost container. Nothing here requires you to build a Docker image; AWS ships a family of images maintained per framework and per version, and you just point an estimator at one.

The Estimator, in one page

The Estimator is the SDK object that describes a training job. It owns four things: what image to run, on what instance, with which hyperparameters, and where to put the result.

import sagemaker
from sagemaker.image_uris import retrieve
from sagemaker.inputs import TrainingInput
from sagemaker.estimator import Estimator

session = sagemaker.Session()
role = "arn:aws:iam::123456789012:role/SageMakerExecutionRole"
bucket = "sagemaker-eu-west-1-123456789012"

image_uri = retrieve(framework="xgboost", region=session.boto_region_name, version="1.7-1")

xgb = Estimator(
    image_uri=image_uri,
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    output_path=f"s3://{bucket}/churn/models/",
    base_job_name="churn-xgboost",
    hyperparameters={
        "objective": "binary:logistic",
        "eval_metric": "auc",
        "num_round": 200,
        "max_depth": 6,
        "eta": 0.1,
        "subsample": 0.8,
    },
)

Nothing has run yet. The job launches only when you call fit, and fit needs to know where the data is.

Input channels: several inputs, several prefixes

SageMaker delivers data to a training container through channels, one prefix per channel. The built-in XGBoost expects at minimum a train channel and, if you want early stopping, a validation channel.

train_input = TrainingInput(
    s3_data=f"s3://{bucket}/churn/processed/train/",
    content_type="text/csv",   # XGBoost built-in still expects CSV
    distribution="FullyReplicated",
)
val_input = TrainingInput(
    s3_data=f"s3://{bucket}/churn/processed/validation/",
    content_type="text/csv",
    distribution="FullyReplicated",
)

xgb.fit({"train": train_input, "validation": val_input}, wait=True)

Two details matter. XGBoost built-in expects CSV with no header and the target as the first column — a constraint that overrides the Parquet advice of module 3 for this specific algorithm; module 5 shows how a custom script frees you from it. And distribution="FullyReplicated" means each training instance receives the full dataset. On more than one instance, "ShardedByS3Key" gives each worker a slice, which is what distributed training expects.

Instance type and the ninety-percent rule

The estimator is billed per second on the instance you asked for, whether the training uses it well or not. Three instance families cover almost every case:

Family	Use	Approximate cost
ml.m5.xlarge	Structured data, XGBoost, scikit-learn	$0.269 / h
ml.c5.4xlarge	Larger structured data, CPU-parallel	$0.816 / h
ml.g4dn.xlarge	Deep learning on a modest GPU	$0.736 / h
ml.p3.2xlarge	Serious deep learning, one V100	$3.825 / h

The ninety-percent rule: if a training job does not use 90 % of the chosen instance for most of its duration, either the instance is oversized or the data pipeline is starving the compute. In the second case, module 5's File versus Pipe input mode matters more than a bigger instance.

Reading the results

fit uploads model.tar.gz to output_path and writes structured logs to CloudWatch. From the SDK you can grab the last-round metrics:

description = xgb.jobs[-1].describe()
metrics = description["FinalMetricDataList"]
for m in metrics:
    print(m["MetricName"], m["Value"])
# validation:auc 0.847
# train:auc 0.902

That gap between train and validation AUC (0.902 versus 0.847) is the module-9 story of course 07: a small amount of overfitting, controllable by lowering max_depth or eta, which module 6 automates.

Spot instances: cheaper, with a catch

Spot instances are unused EC2 capacity, sold at 60 % to 80 % off, that AWS may reclaim with a two-minute notice. On a training job that already saves its state to S3, that is a great trade.

xgb = Estimator(
    ...,
    use_spot_instances=True,
    max_run=3600,          # hard wall-clock cap: 1 hour
    max_wait=7200,         # allow up to 1 hour of waiting for capacity
    checkpoint_s3_uri=f"s3://{bucket}/churn/checkpoints/",
)

The checkpoint_s3_uri is what makes an interruption survivable: SageMaker copies files written to /opt/ml/checkpoints/ inside the container to S3 as they appear. When the job is interrupted and rescheduled, the new instance restores those files at the same path and the script resumes. The built-in XGBoost writes those checkpoints for you every few rounds; a custom script has to opt in explicitly.

Two settings are non-negotiable. max_run caps the total instance time, so a stalled job cannot burn the budget. max_wait bounds how long the job waits for Spot capacity before failing; without it, a training job can sit in the queue for hours during a busy AWS region.

Spot is not free lunch

The savings are real, and so is the risk: a job that has run for fifty-nine minutes and is interrupted at minute sixty pays for the sixty minutes and restarts from the last checkpoint. Without checkpoints, that is the whole training re-run. Spot pays off on long jobs with regular checkpoints, and it is a poor choice for the last-minute hyperparameter tune before a demo.

Summary

• The Estimator describes a job — image, instance, hyperparameters, output; fit launches it and blocks until done.
• Channels map S3 prefixes to training folders; the built-in XGBoost expects CSV, target-first, no header.
• Instance choice follows the ninety-percent rule: undersized starves the compute, oversized wastes money.
• Spot cuts the bill by up to 80 % on jobs that checkpoint; max_run and max_wait prevent the failure modes.

Next module: replace the built-in XGBoost with a scikit-learn script — the more common case in production.