Module 5 — Structured outputs: JSON and schemas

Modules 1 to 4 got the model to produce the right four fields on the running case. The next stage of the pipeline — a support ticketing system — needs those fields as JSON, not as four labelled lines. This module explains how to get JSON that always parses, that respects a schema, and that fails cleanly when the input does not match.

Free-form JSON is not a solution

The obvious first attempt is to change the format instruction:

SYSTEM = """... Return the four fields as JSON with keys
reason, product, urgency, requested_action."""

On ninety-five per cent of calls, this works. On the remaining five per cent, the model returns:

• a JSON object wrapped in a Markdown code fence ( json )
• extra commentary before or after the object
• trailing commas that break json.loads
• keys in a different order — harmless, unless downstream code relies on order
• a missing key when the field was "not stated"

A one-line json.loads in production fails on all of these. Fixing it with regular expressions is a losing battle, because every model update introduces new variations.

JSON mode: parseable, not valid

Most modern APIs offer a JSON mode that guarantees the output is syntactically valid JSON. On the OpenAI API:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": SYSTEM + "\nRespond with a single JSON object."},
        {"role": "user", "content": email},
    ],
    response_format={"type": "json_object"},
    temperature=0,
)
data = json.loads(response.choices[0].message.content)  # always parses

This solves the syntactic problem, but not the semantic one. The model can still return a key you did not ask for, omit one you needed, or put a sentence where you wanted an enumerated value. JSON mode guarantees "the string parses", nothing more.

Structured output with a schema

The stronger mechanism is schema-constrained generation, offered under different names by the major providers. You describe the expected structure once, and the model is forced to produce output that matches:

from pydantic import BaseModel
from typing import Literal

class ComplaintFields(BaseModel):
    reason: str
    product: str
    urgency: Literal["low", "medium", "high"]
    requested_action: str

response = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": email},
    ],
    response_format=ComplaintFields,
    temperature=0,
)
data: ComplaintFields = response.choices[0].message.parsed

Two important properties follow:

• The urgency field can now only contain low, medium or high. The model cannot invent "very urgent" or "asap".
• Missing fields are impossible: the schema requires all four.

The schema is stronger than any prompt instruction, because it constrains generation at the decoding step, not by asking politely.

Handling optional fields and unknown values

Real data has holes. A complaint email may not mention the product name. The right modelling is to declare optional fields explicitly:

from typing import Optional

class ComplaintFields(BaseModel):
    reason: str
    product: Optional[str] = None
    urgency: Literal["low", "medium", "high", "unknown"]
    requested_action: Optional[str] = None

Two design choices matter here:

• product is Optional[str]: absence is encoded as None, not as an empty string or the string "not stated". Downstream code can test for None cleanly.
• urgency gains an explicit "unknown" value rather than being optional. When the email genuinely does not let you decide, you want the model to say so, not to leave the field null and force downstream code to guess whether that means "not extracted" or "not present".

The general principle: make the model tell you what it does not know, do not force it to invent to fill a required slot.

A schema does not verify facts

Schema constraints prevent structural errors, not factual ones. The model can happily return product: "QuietPro X5" when the email said X3. Validation still requires either a lookup against a source of truth or a separate check pass.

The retry loop, done right

Even with schema-constrained generation, a request can fail: network error, provider rate limit, occasional validation refusal on adversarial input. A well-behaved retry loop looks like this:

import time, json
from pydantic import ValidationError

def extract_with_retry(email: str, max_attempts: int = 3) -> ComplaintFields:
    last_error = None
    for attempt in range(max_attempts):
        try:
            response = client.beta.chat.completions.parse(
                model="gpt-4o-mini",
                messages=[
                    {"role": "system", "content": SYSTEM},
                    {"role": "user", "content": email},
                ],
                response_format=ComplaintFields,
                temperature=0,
            )
            return response.choices[0].message.parsed
        except ValidationError as e:
            last_error = e
            time.sleep(2 ** attempt)  # exponential backoff
    raise RuntimeError(f"Failed after {max_attempts} attempts: {last_error}")

Three specifics worth noting:

• Exponential backoff ($2^{n}$ seconds) avoids hammering the provider during a transient outage.
• The temperature stays at zero across retries. Raising it on retry is a common tempting mistake — it turns a deterministic failure into a random one and hides the real problem.
• After max_attempts, the function raises. Silent failure is what makes broken prompts survive in production for weeks.

Cost of validation

Adding schema constraints and Pydantic parsing costs almost nothing at inference time — the constraint is applied during decoding. What does cost is the tighter API tier required by some providers, and the small latency overhead of parsing. On the running case, the difference is below ten milliseconds per call.

The real cost lives elsewhere: writing and maintaining the schema itself. Every added field, every allowed value change, every new optionality decision requires re-evaluating on your test set. That maintenance is the subject of module 9, and it is what makes structured outputs a discipline rather than a one-off setup.

Version the schema like code

The schema is the interface between the model and everything downstream. Give it a version number, bump it on any breaking change, and log which version produced each row of output. When a downstream job breaks in two weeks, the version tells you whether the model changed, the schema changed, or something else entirely.

In summary

• Free-form JSON works on most calls and breaks on the rest; JSON mode gives you parseable strings but does not check keys or values.
• Schema-constrained generation — via Pydantic or an equivalent JSON schema — forces valid structure and enumerated values at the decoding step, stronger than any prompt instruction.
• Model optional fields explicitly and add an "unknown" value where absence is meaningful; a schema does not verify facts, only structure.
• Retry with exponential backoff, keep temperature at zero, and raise after max_attempts — silent failure is what lets broken prompts survive in production.

Next module: controlling how the answer sounds — style, length and tone — with constraints that can actually be measured.