Lesson 1 — Where bias comes from
"The data was biased" is true and too vague to act on. Bias enters at six identifiable points, each with its own remedy, and the most consequential ones are not in the data at all.
1. Historical bias — the data is accurate and the world was unjust
The data faithfully records past decisions, and those decisions were discriminatory. A model trained to reproduce them reproduces the discrimination, with better consistency than the humans managed.
The best-known case is a recruiting tool trained on a decade of hiring decisions at a company whose engineering hires had been overwhelmingly male. It learned to downgrade CVs containing indications of being a woman, including the names of women's colleges. It was abandoned. Notice that the data was not wrong — it was an accurate record of decisions that should not be repeated.
This is the hardest source, because no amount of data cleaning helps. The data is correct; the target is the problem. Addressing it requires deciding you are not trying to reproduce past decisions, which is a change to the objective rather than to the pipeline.
2. Representation bias — who is missing
The training data under-represents some groups, so the model performs worse for them.
Audits of commercial gender classification systems in 2018 found error rates under one percent for lighter-skinned men and over thirty percent for darker-skinned women, traceable directly to the composition of the benchmark datasets. Speech recognition has shown substantially higher word error rates for some dialects. Medical models trained predominantly on one population transfer poorly to others.
Remedy: measure representation before training, collect deliberately where a group is thin, and report performance by group so the gap is visible. The last part matters most — a single aggregate accuracy figure hides exactly this.
3. Measurement bias — you recorded a proxy
The label you can measure is not the outcome you care about, and the gap between them carries bias.
The clearest documented case: a widely deployed healthcare system used to identify patients needing extra care used past healthcare spending as a proxy for health need. Because less money had historically been spent on Black patients at equal levels of illness, the model systematically under-identified them. The model predicted its stated target accurately. The target was wrong.
The same shape recurs constantly: "arrested" as a proxy for "committed a crime", "promoted" as a proxy for "performed well", "clicked" as a proxy for "found valuable", "reported" as a proxy for "occurred".
Remedy: for every label, ask what it actually measures and whose behaviour determined it. This question catches more real problems than any fairness library.
4. Aggregation bias — one model for groups that differ
A single model applied to populations for which the underlying relationships differ will fit the majority and misfit the rest.
Medical examples are the clearest: diagnostic thresholds that differ by ancestry, symptoms that present differently by sex. A model trained mostly on one group encodes that group's relationships as universal.
Remedy: check whether performance differs by subgroup, and consider separate models or group-aware features where it does. This trades against a legitimate concern that group-specific treatment can itself be objectionable, which is a decision rather than a calculation.
5. Deployment bias — used for something it was not validated for
The model works as specified and is then used differently.
A risk score built to prioritise a review queue starts being used to make automatic decisions. A tool validated on one population is deployed on another. A screening model intended to flag cases for attention becomes the final answer because the queue is long.
Remedy: document intended use and its boundaries explicitly, and treat any change of use as requiring revalidation. This is why the intended-use section of a model card is the part that repays the effort.
6. Feedback loops — the model shapes its own future data
Predictions influence what happens, which influences what gets recorded, which trains the next model.
Predictive policing is the canonical example: patrols directed to an area produce arrests there, which become evidence that the area needs patrols, while comparable activity elsewhere goes unrecorded. The same structure appears in fraud detection (only flagged transactions are investigated), in recommendation (users can only click what was shown), and in hiring (only interviewed candidates produce outcomes).
Remedy: deliberate exploration outside the model's recommendations, with those outcomes tracked separately. Without it, confidence grows while the blind spot grows with it.
Why removing sensitive attributes does not work
The intuitive fix — delete ethnicity, sex, age from the features — is called fairness through unawareness, and it fails for two reasons.
Proxies reconstruct the attribute. Postcode correlates with ethnicity in most segregated cities. First names carry information about sex and origin. Education history, purchasing patterns, browsing behaviour, even typing speed all correlate. A model with enough features rebuilds what you removed, and does so invisibly.
You lose the ability to measure. This is the part that gets missed. If you do not record group membership, you cannot compute error rates by group, so you cannot detect that your system discriminates. Removing the attribute removes the evidence, not the bias.
The uncomfortable but correct implication: measuring fairness requires collecting the data you are not allowed to make decisions with. Data protection law generally accommodates this — processing for bias monitoring is a recognised purpose — and it needs to be set up deliberately, with the attribute available to the auditing process and unavailable to the model.
Not "is our data biased?" but: who is likely to be harmed if this system is wrong, and would we find out? The second half matters more. Most documented harms persisted for months or years because nobody was measuring the group that was affected.
In three sentences
Bias enters at six points — historical bias where the data accurately records unjust past decisions, representation gaps, measurement bias where the recorded label is a loaded proxy for what you care about, aggregation bias from one model serving groups whose relationships differ, deployment bias from use beyond what was validated, and feedback loops where predictions shape future training data — and the hardest of these are not fixable by cleaning data. The documented cases are instructive precisely because the models worked as specified: a recruiting tool faithfully reproduced a decade of male-dominated hiring, and a healthcare model accurately predicted spending while systematically under-identifying Black patients whose care had historically cost less. Removing sensitive attributes fails because other features act as proxies and because you then cannot measure whether discrimination is occurring, so responsible practice means collecting group data for auditing while keeping it out of the model.