Skip to main content

Module 9 — Documented successes and famous failures

The field produces two literatures: victory press releases and failure autopsies. Both are instructive — provided you look for the recurring patterns rather than the anecdotes. This module distills them into an analysis grid applicable to any project, including yours.

What durable successes have in common​

The deployments that have created value for years — bank fraud detection, content recommendation, machine translation, industrial predictive maintenance, email sorting — share five traits, with striking regularity.

A narrow, stable task. Classify, rank, estimate: a bounded scope (module 3), a definition of success that does not move every quarter.

Abundant, self-renewing data. The activity itself produces the examples and the labels, continuously: every transaction, every click, every breakdown enriches the training set. No annotation project to repeat.

An organized tolerance for error. None of these systems is perfect; all are designed to be wrong without catastrophe: the bad recommendation is ignored, the suspicious transaction goes to human review, the maintenance alert triggers an inspection rather than a plant shutdown. The error is absorbed by the process.

A human in the right place. Either in the loop (the alert is handled by an analyst) or on the loop (the system acts alone, the human watches the aggregates and can pull the plug).

Continuous improvement. Monitoring, retraining, controlled experiments: the model is a living product, not a deliverable.

The anatomy of famous failures​

The field's public failures are well documented; here are four families, each carried by an emblematic case.

Learned bias, then amplified. A major technology company abandoned its résumé-screening tool after finding it systematically penalized female applications: trained on ten years of past hiring, mostly male, it had learned that "male" correlated with "hired" — down to devaluing the word "women's" in résumés. The pattern: historical data encodes past practices; the model reproduces them consistently and at scale. No cosmetic fix sufficed; the project was shut down.

The toxic feedback loop. A public conversational agent, designed to learn from its interactions, became abusive in less than twenty-four hours under the coordinated action of malicious users. The pattern: a system that learns continuously from its environment can be attacked through that environment. Continuous learning demands guardrails that the launch had not planned for.

The metric optimized against the objective. Recommender systems optimized for watch time favored the most extreme content — not by intent, but because measured engagement is not the intended satisfaction (module 7). Years of fixes followed. The same pattern shows up, in miniature, in any project whose metric is a convenient stand-in for the real objective.

Deployment beyond the domain of validity. Facial recognition systems, accurate on their evaluation sets, proved markedly less reliable on faces under-represented in the training data — with serious consequences in policing contexts. Diagnostic support tools, excellent in the hospital that built them, failed in other institutions with different equipment. The pattern: measured performance only holds on the measured distribution (module 5); the gap between the lab and the field is paid in full.

What all these failures have in common

None is an algorithm failure. They are failures of overall design: unquestioned data, badly chosen metric, missing guardrails, too-narrow validation. The technology worked; the system — in the broad sense, humans and processes included — was badly thought out. That is good news in disguise: these failures are avoided by method, not by genius.

The analysis grid in five questions​

Facing any project — yours, a vendor's, a competitor's — five questions summarize the preceding modules:

  1. The task: is it narrow, stable, defined by a downstream decision? (modules 3 and 8)
  2. The data: where does it come from, what past practices does it encode, does it cover the real field? (modules 6 and 9)
  3. The metric: does it measure the objective or a convenient stand-in? Who priced the two error types? (module 7)
  4. The error: what happens concretely when the system is wrong — and does the organization absorb it? (this module)
  5. The long run: who monitors drift, with what thresholds and what retraining plan? (module 8)

A project that answers all five clearly has the profile of the durable successes. Every vague answer is a named — and therefore manageable — risk.

Key takeaways​

  • Durable successes: narrow task, data renewed by the activity, error absorbed by the process, human well placed, continuous improvement.
  • Famous failures follow four patterns: learned bias, feedback loop, stand-in metric, leaving the domain of validity — all avoidable by method.
  • Failures are failures of overall design, not of algorithms.
  • The five-question grid — task, data, metric, error, long run — can be applied to any project in one meeting.

Last course module: the current limits of the technology, and the research directions that structure the field — to finish with an up-to-date overall picture.