Lesson 5 — Accountability and regulation
A system causes harm. Who answers for it?
The vendor points to their terms. The deploying organisation points to the vendor. The team points to the training data. The operator says they followed the recommendation. Everyone is partially right, and nobody is accountable — which is the actual failure.
The diffusion problem, and the fix
Responsibility spreads across a chain: whoever built the model, whoever assembled the training data, whoever deployed it, whoever operated it, whoever configured the threshold. Each link has a plausible account of why it was not their part.
The fix is not sophisticated: name a person or role accountable for the outcome, before deployment. Not for the model's accuracy — for what happens to people affected by it. That person needs authority to stop the system, budget to fix it, and the information required to know when something is wrong.
Organisations that do this handle incidents. Organisations that do not discover after an incident that responsibility was never assigned, which is a governance failure rather than bad luck.
Human oversight that is real
"A human reviews the output" appears in nearly every governance document and is frequently theatre. Three conditions separate the real thing from the appearance.
Genuine authority. The reviewer can disagree and have their disagreement stand. If overriding requires justification to a manager while agreeing requires nothing, the asymmetry decides the outcome.
Time. A reviewer with four seconds per case is a rubber stamp. Real review requires the workload to be sized for it, which costs money and is the reason it is so often not done.
Information. The reviewer needs what the model saw, what it output, why, and what the confidence means — including the model's known failure modes. Someone shown only a score cannot meaningfully assess it.
Then there is automation bias, which is well documented across aviation, medicine and criminal justice: people defer to a machine recommendation even when their own judgement contradicts it, and they do so more when tired, busy or accountable for disagreeing. Designing against it means presenting the model's uncertainty prominently, recording overrides without penalising them, and occasionally auditing agreement rates — a reviewer who agrees ninety-nine percent of the time is not reviewing.
What regulation now requires
The EU AI Act is the most developed framework and its structure — obligations tiered by risk — is being echoed elsewhere. The tiers:
Prohibited. Some uses are banned outright, including social scoring by public authorities, exploitation of vulnerabilities of specific groups, certain biometric categorisation by sensitive characteristics, and untargeted scraping of facial images to build recognition databases. Worth knowing because "we could technically do this" is not the relevant question.
High risk. Employment and worker management, creditworthiness, access to education, essential public and private services, law enforcement, migration, safety components of regulated products. Requirements include:
- Risk assessment and mitigation, documented, before deployment
- Data governance: provenance, representativeness, known gaps
- Technical documentation of design, performance and limitations
- Logging sufficient to reconstruct a decision
- Human oversight with genuine authority
- Accuracy, robustness and cybersecurity appropriate to the purpose
- Conformity assessment before entering the market
Limited risk. Chatbots and generated content mainly owe transparency: people should know they are dealing with a machine, and synthetic content should be identifiable.
Minimal risk. Most applications. General obligations only.
Beyond the EU, the direction is consistent even where the mechanism differs: sectoral regulators applying existing law to automated decisions, transparency requirements for generated content, and specific rules on biometrics and employment screening. The practical planning assumption is that a high-stakes system will face documentation and audit expectations wherever it operates.
Notice how much of this is engineering you would want anyway
Versioned data and models, prediction logging, subgroup performance metrics, documented intended use, a rollback path, a named owner. That is the MLOps course, not a separate compliance workstream.
Teams that build properly find most compliance documentation falls out of what they already have. Teams that do not, produce documentation retrospectively that describes a system they cannot actually reconstruct — which is both the more expensive path and the one that fails an audit.
A review process, sized to the stakes
Applied uniformly, governance becomes an obstacle people route around. Tier it.
For any system: a named owner, a one-page description of purpose and limitations, and monitoring.
Adding for medium stakes: performance broken down by relevant subgroup, a documented intended-use boundary, a review schedule, and a stated fallback when the system is unavailable.
Adding for high stakes:
| Element | What it means concretely |
|---|---|
| Impact assessment | Who could be harmed, how badly, how would you find out |
| Affected-party consultation | Talk to the people the system decides about |
| Fairness criterion chosen and justified | Lesson 2, decided consciously and recorded |
| Subgroup metrics, re-measured after each retrain | Not once at launch |
| Contestability route | A human with authority, within a stated timeframe |
| Full audit trail | Any decision traceable to model version, inputs and training data |
| Independent review | Someone outside the building team signs off |
| Incident plan | Who is called, who can switch it off, how affected people are told |
| Sunset condition | Under what circumstances this system is withdrawn |
The last row is the one almost nobody writes and the one that most improves decision quality, because agreeing in advance what would make you stop forces clarity about what the system is for.
Questions to raise, and to answer
Whether reviewing someone else's system or your own:
- What decision does this make, and what happens to a person when it is wrong?
- Who is accountable for that outcome, by name?
- What is the performance for each affected group, not overall?
- Which fairness criterion was chosen, and who decided?
- Can an affected person find out a decision was made and challenge it?
- What was this validated for, and how do you prevent use beyond that?
- How would you learn that it had stopped working?
- What would make you switch it off?
A system whose owners cannot answer these is not ready, regardless of its accuracy.
In three sentences
Accountability fails by diffusion — vendor, builder, deployer and operator each have a plausible account of why it was not their part — and the remedy is naming a person accountable for the outcome before deployment, with authority to stop the system and the information to know when something is wrong. Human oversight only counts when the reviewer can override without penalty, has time to think, and sees the inputs, the uncertainty and the known failure modes, because automation bias is well documented and a reviewer agreeing ninety-nine percent of the time is not reviewing. Regulation tiers obligations by risk, prohibiting some uses outright and requiring risk assessment, data governance, documentation, logging, oversight and conformity assessment for high-risk uses — most of which is the engineering you would want anyway, which is why teams that build properly find the documentation falls out of what they already have.
Next — Lesson 6: recap and FAQ →