Module 3 — Fairness metrics and their incompatibilities

Module 2 located the sources of bias. This module answers the next question: how do we measure whether the model is fair, once it is trained? The uncomfortable result is that several intuitive definitions of fairness are mathematically incompatible, so choosing a metric is choosing a value system. This module makes that choice explicit and computes the three main metrics with Fairlearn on the credit scorer.

The three definitions that dominate the literature

Demographic parity (also called statistical parity) requires that the approval rate be equal across groups. If men are approved 71 % of the time, women should be too. Formally, $P(\hat{Y}=1 \mid A=a) = P(\hat{Y}=1 \mid A=b)$ where $A$ is the protected attribute. This definition ignores the true default rate; it enforces equal outcomes at the decision stage, whatever the underlying data.

Equalized odds requires that the model be equally accurate across groups on both classes. Concretely: the true positive rate and the false positive rate must be equal across groups. A false rejection should be equally likely regardless of gender, and so should a false approval. Formally, $P(\hat{Y}=1 \mid Y=y, A=a) = P(\hat{Y}=1 \mid Y=y, A=b)$ for $y \in \{0, 1\}$. This definition takes the label at face value.

Calibration requires that when the model predicts a probability of default of, say, 15 %, that prediction should be right at 15 % across groups. Among all applicants scored 0.15, the actual default rate should be 15 % for men and 15 % for women. Formally, $P(Y=1 \mid \hat{S}=s, A=a) = s$ for every group $a$ and score $s$. This is the definition banks intuitively adopt because it corresponds to a scoring model that "means the same thing" everywhere.

The impossibility result

Chouldechova (2017) and Kleinberg, Mullainathan and Raghavan (2016) proved a result that reshaped the field. When the base rate — the actual default rate — differs between groups, no non-trivial classifier can satisfy calibration and equalized odds simultaneously. If two groups have different underlying default rates, forcing equal error rates breaks calibration, and enforcing calibration guarantees unequal error rates.

The intuition is direct. Suppose 10 % of group A actually defaults but 20 % of group B does. A perfectly calibrated model that assigns each applicant their true risk will, on average, over-predict for group A and under-predict for group B, hence produce different false positive rates. Conversely, if we enforce equalized odds by re-thresholding, applicants scored 0.15 in group A will have a different real default rate than those scored 0.15 in group B — calibration is gone.

Two consequences follow. First, the fairness debate is not resolvable by a better algorithm. Second, choosing a metric is a normative decision about which error type matters most for which group. That decision belongs to a human committee, not to the data scientist.

Fairlearn on the credit scorer

Fairlearn is Microsoft's open-source library for measuring and mitigating disparities in scikit-learn-compatible pipelines. The audit computes disparities with MetricFrame:

from fairlearn.metrics import (
    MetricFrame, selection_rate,
    true_positive_rate, false_positive_rate,
    demographic_parity_difference, equalized_odds_difference,
)
from sklearn.metrics import accuracy_score

y_true = df["defaulted"]
y_pred = model.predict(X)
gender = df["gender"]

metrics = {
    "accuracy": accuracy_score,
    "selection_rate": selection_rate,
    "tpr": true_positive_rate,
    "fpr": false_positive_rate,
}

mf = MetricFrame(
    metrics=metrics,
    y_true=y_true,
    y_pred=y_pred,
    sensitive_features=gender,
)

print(mf.by_group)
print("DP diff:", demographic_parity_difference(y_true, y_pred, sensitive_features=gender))
print("EO diff:", equalized_odds_difference(y_true, y_pred, sensitive_features=gender))

On the credit dataset with a threshold at 0.5, by_group shows a selection rate of 0.71 for men versus 0.63 for women (demographic parity difference: 0.08), a true positive rate of 0.82 versus 0.74 (equalized odds violation dominated by the TPR gap), and an accuracy that is nearly equal at 0.86 versus 0.85. The model is more likely to correctly identify a solid male applicant than a solid female one — an "opportunity" disparity that the accuracy score entirely hides.

Mitigations and their costs

Fairlearn ships three families of mitigation. Pre-processing reweighs training examples so that each subgroup contributes equally. In-processing (ExponentiatedGradient) trains the model under a fairness constraint. Post-processing (ThresholdOptimizer) picks a different decision threshold per group to equalize the chosen metric on a held-out set. All three trade some accuracy for reduced disparity. On the credit scorer, ThresholdOptimizer targeting equalized odds cuts the TPR gap from 0.08 to 0.02 at the cost of 1.5 accuracy points. That trade is a business decision, documented in the model card of module 6.

Per-group thresholds are technically effective and legally delicate. In several jurisdictions, applying a different rule to two people based on a protected attribute is itself discrimination, even when the intent is corrective. The audit must record this and let the legal team decide, not decide for them.

What "fair" means for this bank

At the end of this module, the credit-scoring audit records the following: the risk committee has chosen equalized odds as the primary fairness metric, arguing that a good applicant should have the same chance of being approved regardless of gender. It has accepted a maximum equalized odds difference of 0.03 across gender and 0.05 across three age bands. It has not chosen demographic parity, because the base default rate genuinely differs across age bands and mechanically equalizing approvals would push young, high-risk borrowers into loans they cannot repay. The impossibility theorem means the model will not be calibrated across those same groups; the model card must say so, and the recourse procedure of module 7 must account for it.

Summary

• Demographic parity, equalized odds and calibration are three intuitive but non-equivalent fairness definitions.
• The impossibility theorem says that when base rates differ, no non-trivial classifier satisfies calibration and equalized odds at the same time.
• Choosing a fairness metric is a values decision, made by a committee and written into the model card, not a technical shortcut.
• Fairlearn computes disparities and offers pre-, in- and post-processing mitigations; per-group thresholds are effective and legally sensitive.

Next module: making a decision explainable with SHAP and LIME, and why an explanation is not the same thing as a justification.