Skip to main content

Module 9 — Vertex AI Pipelines

The previous eight modules built the fraud pipeline as separate notebook cells. That works for one run; it breaks the moment the whole flow must run every week, reproducibly, with someone else's understanding. This module wires the whole thing — pull data, train, tune, evaluate, register, deploy — into a Vertex AI Pipeline written in KFP v2, with automatic lineage and a deployment gate.

What a pipeline actually buys​

The value of a pipeline is not "orchestration" in the abstract. It is three concrete properties:

  • Reproducibility. A pipeline run is a container graph with pinned versions; running it again in six months produces the same artifacts.
  • Lineage across steps. Vertex records every artifact input and output; the "which dataset produced which model" question of module 6 becomes a query on a graph, not archaeology.
  • Gates. A step can consume the evaluation metric of a previous step and refuse to promote a bad model. This is what turns a script into a release process.

Components: the atomic unit​

A component is a container that reads inputs, does one thing, and writes outputs. In KFP v2, the easiest form is a lightweight Python component:

from kfp import dsl
from kfp.dsl import component, Input, Output, Dataset, Model, Metrics

@component(
    base_image="python:3.11-slim",
    packages_to_install=["google-cloud-bigquery==3.25.0", "pyarrow==15.0.0"],
)
def export_training_slice(
    project: str,
    query: str,
    output: Output[Dataset],
) -> None:
    from google.cloud import bigquery
    client = bigquery.Client(project=project, location="europe-west1")
    df = client.query(query).to_dataframe()
    df.to_parquet(output.path, index=False)
    output.metadata["rows"] = len(df)
    output.metadata["fraud_rate"] = float(df["is_fraud"].mean())

Two properties are worth naming. output.path is a local path Vertex has bound to a GCS artifact — the pipeline reads it back for the next step and stores it permanently under the run ID. output.metadata["rows"] becomes queryable metadata attached to that artifact, useful for later analysis.

Composing components into a pipeline​

The pipeline itself is a Python function decorated with @pipeline, calling components as if they were functions. The tool that would take a full week the first time reads, in the end, like a top-level script:

@dsl.pipeline(
    name="fraud-training-pipeline",
    pipeline_root="gs://fraud-detection-dev-vertex-eu/pipelines",
)
def fraud_pipeline(
    project: str,
    region: str,
    training_query: str,
    baseline_pr_auc: float = 0.80,
):
    slice_op = export_training_slice(project=project, query=training_query)

    train_op = train_xgb(
        training_data=slice_op.outputs["output"],
        n_estimators=800,
        max_depth=6,
    ).set_memory_limit("16G").set_cpu_limit("8")

    eval_op = evaluate_model(
        model=train_op.outputs["model"],
        validation_data=slice_op.outputs["output"],
    )

    with dsl.If(eval_op.outputs["pr_auc"] >= baseline_pr_auc,
                name="promote-if-better-than-baseline"):
        upload_op = upload_to_registry(
            model=train_op.outputs["model"],
            evaluation=eval_op.outputs["metrics"],
            display_name="fraud-xgb",
        )
        deploy_op = deploy_to_endpoint(
            model=upload_op.outputs["model_resource"],
            endpoint_display_name="fraud-xgb-endpoint",
            traffic_percentage=10,       # challenger, per module 7 convention
        )

The dsl.If block is the deployment gate: no upload, no deployment, if PR-AUC on the validation split is below the baseline. That single conditional prevents the class of "we shipped a model that was worse than the one already in production" incidents that most teams learn from painfully.

Compilation and submission​

A KFP pipeline is not run directly. It is compiled to a JSON specification that Vertex reads:

from kfp import compiler

compiler.Compiler().compile(
    pipeline_func=fraud_pipeline,
    package_path="fraud_pipeline.json",
)

pipeline_job = aiplatform.PipelineJob(
    display_name="fraud-training-2026-09-06",
    template_path="fraud_pipeline.json",
    pipeline_root="gs://fraud-detection-dev-vertex-eu/pipelines",
    parameter_values={
        "project": "fraud-detection-dev",
        "region":  "europe-west1",
        "training_query": TRAINING_QUERY,
        "baseline_pr_auc": 0.82,
    },
    enable_caching=True,
)

pipeline_job.run(service_account="vertex-pipelines@fraud-detection-dev.iam.gserviceaccount.com")

Two properties from that block.

enable_caching=True. When a step's inputs (component version, parameters, upstream artifacts) are unchanged from a previous run, Vertex reuses the output rather than re-executing. On the fraud pipeline, this means "iterating on the deploy step" does not re-export data or retrain the model — a five-minute iteration instead of a two-hour one.

A dedicated service account for pipelines. vertex-pipelines@... gets the union of the permissions the components need: BigQuery read on raw.transactions, GCS read/write on the pipeline root, aiplatform.user on the project. Not the default Compute Engine identity.

Artifacts and lineage: the payoff​

Every artifact a component writes to Output[...].path is stored under pipelines/<run_id>/<component>/output and catalogued in the Vertex metadata store. The Lineage tab of the pipeline UI shows a graph like:

BigQuery query -> Dataset (12M rows, fraud_rate 0.0021)
                    |
                    v
                Trained model (xgb, aucpr 0.86, run 2026-09-06)
                    |
                    v
                Model v7 in registry -> Endpoint fraud-xgb-endpoint (10%)

Six months later, an auditor asks "which query produced version 7?". The answer is a click, not a Slack search. Every deployment to production goes through this graph or it is not a deployment.

Scheduling: the last piece​

A pipeline that runs when someone remembers is not a training pipeline. Vertex Pipelines has a native scheduler:

from google.cloud import aiplatform_v1

client = aiplatform_v1.ScheduleServiceClient(client_options={
    "api_endpoint": "europe-west1-aiplatform.googleapis.com"
})

schedule = client.create_schedule(
    parent="projects/…/locations/europe-west1",
    schedule=aiplatform_v1.Schedule(
        display_name="fraud-weekly-retrain",
        cron="TZ=Europe/Paris 0 4 * * MON",
        create_pipeline_job_request=aiplatform_v1.CreatePipelineJobRequest(
            parent="projects/…/locations/europe-west1",
            pipeline_job=pipeline_job.to_dict(),
        ),
    ),
)

Monday 4 a.m. Paris time, the pipeline runs, and either promotes a new challenger to 10 % or logs "did not beat baseline". This is what "MLOps on Vertex" looks like when it is real.

Common pitfalls to sidestep​

A component that "downloads" its input. Do not gsutil cp inside a component — Vertex has already localised input.path for you. Reading input.path directly is what makes the artifact lineage work.

A pipeline parameter that shadows a component parameter. If the pipeline signature and the component signature both use region, KFP passes the pipeline value; the component's default is ignored. Name pipeline parameters distinctively (region_train, not region) when there is any ambiguity.

Skipping enable_caching on iterating. Debugging a pipeline without caching wastes 15 minutes per attempt. Turn caching on during development; turn it off on the scheduled production run so the training actually re-executes on fresh data.

"The container image is too large" hides real errors

KFP components package their sources into the container image at run time when the base image is generic. When the base image is 1.5 GB and the component reinstalls torch, cold start becomes minutes. Use a prebuilt custom image on Artifact Registry for expensive components — the same rule as module 4 applied to pipelines.

In summary​

  • A Vertex AI Pipeline is a DAG of KFP components — each a container reading typed inputs and writing typed outputs — compiled to JSON and submitted as a PipelineJob.
  • The payoff is reproducibility, lineage across steps and gates; a dsl.If block on the evaluation metric turns "we hope this model is better" into an enforceable promotion rule.
  • Enable caching during development, disable it on scheduled production runs, and give the pipeline its own least-privileged service account.
  • Vertex's built-in Schedule resource replaces Cloud Scheduler + Cloud Function for pipelines — the retrain job becomes a first-class resource with its own history.

Next module: leaving classical ML behind for a moment — calling a foundation model from Model Garden to classify the free-text dispute comments attached to the flagged transactions.