Module 4 — Chain of thought and step decomposition

Module 3 taught the model with examples. On the running case, four fields out of four are now extracted correctly on most emails. "Urgency" is the last one where errors persist: a customer who says "I need this sorted" is high or medium depending on context the model has to weigh across several sentences. This module tests two levers to lift that: thinking out loud, and splitting the task into smaller model calls.

Chain of thought: why and when

Chain-of-thought (CoT) prompting asks the model to produce its reasoning before the answer. The mechanism is simple: on tasks that require several logical steps, letting the model write the intermediate steps into its own context gives it more "computation" to work with. The next token is no longer conditioned on the question alone, but on the question plus its own reasoning trace.

The zero-shot trigger phrase everyone knows is:

CoT_HINT = "Think step by step, then give the final answer."

Added to the system prompt, this triples answer length and, on arithmetic or multi-step tasks, meaningfully improves accuracy.

When it helps

The tasks where CoT lifts scores share three features:

• The answer requires combining several pieces of information from the input.
• The reasoning path is not obvious in a single glance.
• The final answer is verifiable from the reasoning, so the model is unlikely to write correct reasoning and then a wrong conclusion.

Word problems, multi-hop question answering, non-trivial classification with conflicting cues, and yes, our "urgency" decision all qualify.

For urgency on the running case, the promising variant is:

SYSTEM = """You extract four fields from customer complaint emails.
For the "urgency" field, before deciding, list explicitly:
- any dated deadline mentioned;
- any explicit urgency word;
- the customer's stated tolerance ("no rush", "when possible").
Then output the four labelled lines.
"""

The reasoning is not free-form: it is scaffolded with the exact cues the model should surface. Loose "think step by step" prompts help, but a targeted scaffold helps more, because it forces the model to consider the cues you know are decisive.

When it hurts

Chain of thought is not universally beneficial. On these tasks it degrades performance:

• Simple pattern-matching tasks, where the answer is obvious and extra reasoning invents constraints that were not there.
• Format-critical outputs, where the reasoning prose sneaks into the final output despite instructions to separate them.
• Tasks with a clear numeric threshold, where the model may argue itself out of the correct decision by weighing irrelevant nuances.

The failure mode is subtle: accuracy drops by two or three points, well within noise on twenty test inputs. Only a proper evaluation set detects it, which is why module 9 insists on measuring rather than assuming.

CoT is not a general upgrade

"Adding think step by step never hurts" is folklore, and it is wrong. The safe rule is to A/B-test the CoT variant against the direct one on your actual test set and keep whichever wins. If they tie, prefer the shorter one — it costs less and is faster.

Self-consistency: several traces, one vote

A refinement called self-consistency samples the model several times at non-zero temperature, each producing a different reasoning trace, and takes the majority vote on the final answer:

from collections import Counter

def self_consistent(email, k=5):
    answers = []
    for _ in range(k):
        r = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "system", "content": SYSTEM},
                      {"role": "user", "content": email}],
            temperature=0.7,
        )
        answers.append(extract_urgency(r.choices[0].message.content))
    return Counter(answers).most_common(1)[0][0]

On ambiguous inputs, several traces can reach the same conclusion via different paths, which is stronger evidence than a single trace. The cost is $k$ times the calls, so it makes sense on inputs you already know are hard rather than on every input.

Decomposing into sub-calls

Instead of asking one model call to do everything, you can split the task into a sequence of smaller calls, each specialised:

1. Call one: extract the deadline mention, if any, and produce a short canonical form ("2026-09-10" or "none").
2. Call two: extract the requested action.
3. Call three: given the deadline and the action, decide the urgency.

Each sub-call has a smaller decision surface, is easier to evaluate, and can be replaced independently. The trade-off is obvious: three calls cost three times as much as one and add three times the latency.

Decomposition pays off when:

• At least one sub-task is hard on its own and benefits from a dedicated prompt.
• Sub-tasks are reusable across prompts (extracting a date is useful in many pipelines).
• You need to log intermediate decisions for audit or debugging.

Reasoning-model shortcuts

A new generation of models — often called "reasoning models" — is trained to produce a long internal reasoning trace before answering, without you having to trigger it. On these models, adding an explicit "think step by step" instruction is at best redundant and at worst counter-productive: it makes them write less of their own reasoning because they think you want a shorter answer.

The practical guidance:

• On a standard chat model, explicit CoT or scaffolded reasoning can help, tested per task.
• On a reasoning model, prefer minimal, direct instructions and let the model manage its own thinking budget. Long meta-instructions compete with its trained behaviour.

The right length is what the eval set proves

Length of reasoning, number of samples for self-consistency, number of sub-calls: all three are dials. Do not turn them by feel. Run the variant on the twenty-input mini-eval you built for module 3 and read per-field accuracy — that is the only fair verdict.

In summary

• Chain of thought works when the answer requires combining several cues and the reasoning path is not obvious; a scaffolded CoT that lists the cues to check beats a loose "think step by step".
• CoT can hurt on simple pattern-matching or format-critical tasks: always A/B-test against the direct variant and prefer the shorter one on ties.
• Self-consistency votes across several traces and helps on ambiguous inputs; decomposition into sub-calls helps when sub-tasks are hard, reusable or auditable.
• On reasoning models, explicit CoT is often redundant or harmful; prefer minimal instructions and let the model manage its own thinking budget.

Next module: turning the free-text output into structured JSON that the next stage of your pipeline can actually parse.