Module 2 — Sources of bias: collection, labeling, history
The word "bias" carries two meanings that must not be confused. Statisticians use it for the systematic gap between an estimate and the quantity it is meant to estimate. Ethicists use it for a decision that unfairly disadvantages a group. A model can be statistically unbiased and socially biased, and vice versa. Both meanings matter in an audit, and both come from the data, not from the algorithm.
This module opens the credit dataset introduced in module 1 and shows, one type at a time, how bias entered before the first line of Python was written.
The five origins that account for almost everything
Selection bias appears when the training set does not represent the population the model will serve. The bank only has labels for applicants who were approved in the past twelve years — for the rejected ones, no default is observed. The model therefore learns from a truncated sample where the hardest cases are structurally absent. Any group historically rejected more often than others is underrepresented in the training set and worse modeled going forward. This is not a labeling error; it is the shape of the sample.
Measurement bias occurs when the same underlying quantity is recorded differently across groups. Reported "monthly income" is more reliable for salaried employees than for informal workers, gig workers or the self-employed. Two applicants with identical actual income can appear to the model as very different risks because the feature that captures their income is not equally precise. The model does not know it is being fed a noisier signal for one group.
Label bias shows up when the target variable itself reflects a biased process. The label here is "defaulted within twenty-four months", but a default is triggered by a bank action (declaring the loan non-performing), not by a customer intent. If in the past the collections department was quicker to declare defaults for some ZIP codes, the labels themselves carry that pattern. The model learns to reproduce the collection department's historical prejudice, encoded as ground truth.
Feedback loops arise once the model is deployed. A model that under-approves applicants from neighborhood A produces fewer neighborhood-A loans, hence fewer neighborhood-A default observations, hence less data to correct its own error next year. Each retraining reinforces the pattern it created. The system's fairness metric degrades over time even though nobody touched the code.
Proxy variables are features that are ostensibly neutral but strongly correlated with a protected attribute. ZIP code correlates with ethnicity in most cities. Length of the previous employer's name correlates with immigration status. The first name is a near-perfect predictor of gender in most Western datasets. Removing the protected attribute from the features does not remove the discrimination — it merely hides which lever the model is pulling.
A worked example on the credit dataset
The dataset has 120 000 rows. Fifty-two percent of applicants are men, forty-eight percent women. Approval rates are 71 % for men and 63 % for women. That eight-point gap has to come from somewhere; the audit's first job is to locate it.
Fit a logistic regression on the features excluding the gender column, then check how well it predicts gender. If the model reaches an area-under-ROC of, say, 0.79, then the remaining features contain almost all the information gender did. The bank has not removed the signal, it has laundered it. In our data, the top three predictors of gender turn out to be occupation category, part-time employment share and address in a specific set of city districts. Removing occupation drops the AUC to 0.68; keeping it stable requires accepting that "occupation" carries a component of protected information.
This diagnostic — training a probe model to recover the protected attribute from the "cleaned" features — is the fastest way to expose proxies and belongs in every audit. It answers the question that legal teams keep asking: "we removed gender, are we safe?" Almost always: no.
Where each source enters the credit pipeline
| Stage | Source of bias | Concrete symptom on the credit dataset |
|---|---|---|
| Data collection | Selection bias | Only approved applicants have a default label; rejected ones are missing |
| Data collection | Measurement bias | Income declared by self-employed applicants is noisier than for salaried ones |
| Label assignment | Label bias | Collections department historically triggered defaults faster in some ZIP codes |
| Feature engineering | Proxy variables | ZIP code, occupation and first name recover most of the removed gender signal |
| Deployment | Feedback loop | Under-approving neighborhood A yields fewer future observations for that neighborhood |
Each row leads to a distinct mitigation. Selection bias suggests reweighting or importance sampling. Measurement bias suggests collecting confidence intervals on features, not point estimates. Label bias may require re-auditing historical decisions before retraining. Proxies require the probe test above, plus a decision about which features to keep despite their proxy character. Feedback loops require intervention on the deployed policy itself — for example, a small fraction of exploration approvals to keep the training signal alive.
The trap of "the algorithm is neutral"
The most dangerous sentence in an audit meeting is: "the model just learns what is in the data, so if there is a disparity, that is society's problem." It is dangerous because it is half true. The disparity does come from society, but the model amplifies it — through selection, through feedback, and because a metric optimized on biased labels moves the whole system in the direction of those labels. Deploying a model that reproduces the historical rejection rate for women is a decision, not a discovery. Somebody chose the label, chose the features, chose the threshold, and chose to ship. Module 3 will show that even the choice of fairness metric is a values statement dressed as a technical one.
Summary
- Bias enters through collection (who is in the sample), measurement (how features are recorded), labels (how the target was defined), proxies (features that carry protected information) and feedback loops (deployment reshapes future data).
- Removing the protected attribute does not remove discrimination; a probe model trained to predict the attribute from the remaining features reveals how much information leaks through.
- The eight-point approval gap on the credit dataset comes largely from occupation, part-time status and address — not from the removed gender column.
- "The algorithm is neutral" is false in operation: choices at each pipeline stage amplify or dampen the underlying disparity.
Next module: how to measure fairness quantitatively, and why several reasonable definitions are mathematically incompatible.