Lesson 2 — Fairness cannot be satisfied fully
This is the most important technical result in the ethics of machine learning, and it is routinely absent from discussions that need it: the main definitions of fairness are mathematically incompatible. You cannot satisfy them all. You have to choose, and the choice is a value judgement that cannot be delegated to a library.
Four reasonable definitions
Each of these sounds like what fairness means, and they conflict.
Demographic parity. The system selects members of each group at the same rate. If twenty percent of one group are approved, twenty percent of another are too.
Appeals when the base rates in your data are themselves the product of injustice, so matching them would perpetuate it. Objection: it may require selecting less qualified candidates from one group, which people reasonably find objectionable, and it can harm the group it intends to help if the selected members then fail.
Equal opportunity. Among people who genuinely qualify, the same proportion is selected from each group. Equal true positive rates.
Appeals when you care about not missing qualified people from any group. Objection: says nothing about false positives, so one group can be wrongly selected far more often.
Equalised odds. Both true positive and false positive rates are equal across groups. Stronger, and correspondingly harder to achieve.
Appeals when both kinds of error are consequential. Objection: usually costs overall accuracy, and may require explicitly different treatment by group.
Calibration. A predicted score means the same thing regardless of group. If the model says seventy percent, seventy percent of those people experience the outcome, in every group.
Appeals when scores are used by humans making decisions, because otherwise the number is not comparable. Objection: perfectly calibrated scores can still produce very different error rates between groups.
The impossibility result
Formal work published around 2016 established that when base rates genuinely differ between groups, calibration and equalised odds cannot both hold, except in degenerate cases such as a perfect classifier.
This is not an engineering limitation awaiting a better method. It is arithmetic. If the underlying prevalence of an outcome differs between two groups and your scores are calibrated for both, the error rates will differ; if you force the error rates to match, the scores stop meaning the same thing.
So "make the model fair" is not a well-defined instruction. Fair in which sense is a question about which harm you consider worse, and it belongs to whoever is accountable for the decision rather than to whoever implements it.
The case that made this concrete
A 2016 investigation into a recidivism risk tool used in United States courts reported that Black defendants were roughly twice as likely to be incorrectly flagged as high risk, while white defendants were more likely to be incorrectly flagged as low risk.
The vendor responded that the tool was calibrated: a given score corresponded to the same reconviction rate regardless of race.
Both claims were true. They were measuring different fairness criteria, and because the underlying arrest rates differed between groups, satisfying one guaranteed violating the other. The dispute was not about the mathematics — it was about which fairness criterion should govern a decision affecting someone's liberty, and that is a question for a legislature and a court rather than for a data science team.
That is the pattern to recognise. When two parties disagree about whether a system is fair and both have numbers, they are usually both right about different criteria.
How to choose in practice
Since you must choose, choose deliberately and write down why.
Ask what the errors cost, per group. A false positive in fraud screening is an inconvenience; in criminal justice it is a person detained. A false negative in cancer screening is a missed diagnosis; in advertising it is a lost sale. The asymmetry usually points clearly at which criterion matters.
Ask whether the base rate difference is itself the injustice. If one group's lower approval rate reflects historical exclusion rather than genuine difference, matching rates has a strong argument. If it reflects something real and relevant, forcing parity may harm people.
Ask who bears the cost of each choice. The group with worse error rates under your chosen criterion is bearing a real cost, and someone should have decided that consciously.
Involve the affected people. Consultation is not decoration here. Affected communities routinely identify harms that the designing team did not consider, and they have standing to weigh trade-offs that the team does not.
Document the decision. Which criterion, why, who approved it, what the measured gaps are. This is what makes the choice reviewable rather than accidental, and it is increasingly what regulation expects.
What you can do technically
Given a chosen criterion, three points of intervention:
Before training. Reweight or resample the training data, or correct labels where measurement bias is identified. Cheapest and it addresses the cause rather than the symptom.
During training. Add a fairness constraint or penalty to the objective. Effective and it makes the trade-off against accuracy explicit, which some find uncomfortable and is actually a virtue.
After training. Adjust thresholds per group so the chosen criterion holds. Simple and effective, and note that different thresholds by group is explicit differential treatment, which may be legally constrained in your jurisdiction — worth checking before implementing.
Open tooling exists for measuring and mitigating along these lines. None of it chooses your criterion, which is the part that actually matters.
Ignoring fairness entirely, and treating it as a metric to optimise so a library can settle it. The second is more insidious, because it produces a number that looks like diligence while the substantive choice — which harm is worse — was never consciously made by anyone accountable.
In three sentences
Several reasonable definitions of fairness — equal selection rates, equal true positive rates, equal error rates overall, and scores that mean the same thing across groups — conflict, and formal results show that calibration and equalised error rates cannot both hold when base rates genuinely differ between groups. This is arithmetic rather than an engineering gap, which is why the recidivism tool dispute had both sides correct simultaneously: the investigation measured error rates and the vendor measured calibration, and satisfying one guaranteed violating the other. So the criterion must be chosen deliberately by whoever is accountable, based on what each type of error costs each group, whether the base rate difference is itself the injustice, and consultation with affected people — with the choice and its measured gaps documented, because treating fairness as a metric a library optimises produces a number that looks like diligence while nobody made the actual decision.