Module 10 — Foundation models and Model Garden

The fraud-detection pipeline built over the last nine modules works on numeric features of a payment. Some signal, though, lives in text — the dispute comments users write when they contest a charge: "I never used my card at this merchant", "double charge, my child clicked twice", "I asked for a refund three weeks ago and nothing". Classifying these into structured categories fits a foundation model far better than a hand-crafted classifier. This module uses Vertex's Model Garden to do exactly that.

What Model Garden actually is

Model Garden is Vertex's catalogue of foundation models. It lists three families:

• Google's first-party models: the Gemini family (gemini-1.5-pro, gemini-1.5-flash), the older PaLM 2 series, Imagen for image generation, Codey for code.
• Third-party partner models: Anthropic's Claude, Meta's Llama, Mistral's models, served through the same Vertex API.
• Open-source models: Gemma, Llama, Mistral, Falcon — deployable as a Vertex endpoint from the catalogue in one click.

The value of Model Garden over calling model APIs directly is uniformity: same authentication (project + service account), same billing surface, same region, same audit trail. For an organisation already on Vertex, adding a foundation model is not a new vendor onboarding, it is a couple of lines of code.

Calling a Gemini model in ten lines

For the dispute-comment classification, gemini-1.5-flash is enough — it is the fast, cheap model in the family, and this is a short-text classification task where the flagship pro model would be overkill.

from vertexai.generative_models import GenerativeModel
import vertexai

vertexai.init(project="fraud-detection-dev", location="europe-west1")

model = GenerativeModel("gemini-1.5-flash")

def classify_dispute(comment: str) -> str:
    prompt = f"""
    Classify the following payment dispute comment into exactly one category:
    - unauthorised_transaction
    - duplicate_charge
    - refund_not_received
    - product_not_delivered
    - subscription_cancellation
    - other

    Reply with only the category name, nothing else.

    Comment: {comment}
    """
    resp = model.generate_content(prompt, generation_config={"temperature": 0.0})
    return resp.text.strip()

Two choices in that snippet reflect the specific task.

temperature=0.0. A classifier needs the same answer for the same input. Zero temperature makes the response deterministic (up to floating-point noise): the same comment always maps to the same category.

The category list in the prompt. The model does not know the six categories the fraud team uses — it must be told. A closed list plus "reply with only the category name" reduces the failure mode to one: the model returns something not on the list, which the caller catches and routes to other.

From a script to a Vertex endpoint for open-source models

Gemini is called over Vertex's shared surface — no endpoint of your own. For an open-source model — Llama 3.1 8B, say — the pattern of module 7 comes back: you deploy the model from Model Garden to your own endpoint. In the console, that is a "Deploy" button on the model card; in code:

from google.cloud import aiplatform

model = aiplatform.Model("publishers/meta/models/llama3_1@meta-llama-3.1-8b-instruct")

endpoint = model.deploy(
    machine_type="g2-standard-12",
    accelerator_type="NVIDIA_L4",
    accelerator_count=1,
    min_replica_count=1,
    max_replica_count=2,
    endpoint_display_name="llama-dispute-classifier",
)

Now the endpoint is yours — same access logging, same autoscaling, same billing account. The compute is what you pay for; there is no per-token fee on top for models you self-host.

Choose self-hosting when cost matters at high volume (a g2-standard-12 with one L4 costs about $1.20/hour, or roughly $900 a month for one always-on replica — cheaper than one million pro-model tokens at market rate) and when the data must not leave your VPC. Choose the shared API for low volume or when the workload benefits from Gemini's specific strengths.

Managed tuning: adapting a foundation model to your task

Ten labelled examples of dispute comments might already push accuracy from 82 % to 91 % — but only for the specific way your users write. Managed tuning in Vertex is the platform's supervised fine-tuning offering: you upload a JSONL of input/output pairs, Vertex runs the tuning on Google's infrastructure, and produces a tuned model you call through the same API.

The dataset shape:

{"input": "I never used my card at Ikea Toulouse", "output": "unauthorised_transaction"}
{"input": "Charged twice for the same Netflix subscription on 12 Aug", "output": "duplicate_charge"}
{"input": "Cancelled the gym on 2 July, still being billed", "output": "subscription_cancellation"}

Launching a tuning job:

from vertexai.tuning import sft

tuning_job = sft.train(
    source_model="gemini-1.5-flash-002",
    train_dataset="gs://fraud-detection-dev-vertex-eu/tuning/dispute_train.jsonl",
    validation_dataset="gs://fraud-detection-dev-vertex-eu/tuning/dispute_val.jsonl",
    tuned_model_display_name="gemini-flash-dispute-v1",
    epochs=3,
)

Two rules for supervised tuning that experience keeps repeating.

Do not tune on fewer than 100 examples. Below that, the tuned model overfits the exact wording of the training set and behaves worse than the base model on the very next unseen comment. Between 500 and a few thousand examples is the productive range for short-text classification.

Keep a validation set the tuning does not see. The tuning job reports metrics on the training data, which are unfailingly optimistic. The validation set is what you use to decide whether the tuned model actually beats the base model on the population you care about.

Cost per token: the number that decides everything

Foundation-model costs are quoted per million tokens — for text, roughly 750 000 words. The pricing has an input rate (tokens in the prompt) and an output rate (tokens in the response). Output rate is typically 3× to 5× higher than input rate.

Order-of-magnitude figures for Vertex (approximate, check the pricing page):

Model	Input / M tokens	Output / M tokens
gemini-1.5-flash	$0.075	$0.30
gemini-1.5-pro	$1.25	$5.00
claude-3-5-sonnet@vertex	$3.00	$15.00

For the dispute classification: each call is roughly 200 input tokens (prompt + short comment) and 5 output tokens (the category name). At gemini-1.5-flash rates, one call is about $0.00002. Scoring 100 000 disputes a month costs about $2. The cost of deploying the classifier is now dominated by the engineer's afternoon spent writing the prompt, not the token bill.

The same reasoning goes the other way. Sending a 10 000-token document per call to a summarisation service on gemini-1.5-pro costs about $0.0125 per call; 100 000 documents costs $1 250. Cost per token, multiplied by the volume, is the number that decides the model choice.

Prompt length dominates cost, not model choice

Halving the prompt length halves the input cost on every call, on any model. Trimming boilerplate ("Please carefully consider the following..."), removing redundant examples once the model performs, and moving fixed instructions into a system prompt Vertex caches — these three habits typically cut token bills by 30 to 50% at zero quality change. Change prompts before you change model tiers.

In summary

• Model Garden unifies Google's own models, partner models and open-source models under the same Vertex authentication, region and billing — a couple of lines added to an existing project rather than a new vendor.
• Call Gemini with temperature=0 and a closed category list for classification; deploy an open-source model to your own endpoint when cost or data residency requires it.
• Managed tuning adapts a foundation model to your task with 500 to a few thousand input/output pairs; below 100 examples, the tuned model overfits and gets worse.
• Cost per token — input plus output — multiplied by volume is the deciding number; halving prompt length is the cheapest optimisation, and it does not touch model quality.

Next: the recap and 40-question exam that closes the course.