Module 5 — Privacy: anonymization and differential privacy

The credit model was trained on 120 000 real people. That dataset, its derivatives, and — in a subtler way — the trained model itself all carry information about those people. This module covers the two questions that follow. How do we release or share data without letting a third party recover individual identities? And can a machine learning model, all by itself, leak information about the individuals it was trained on? The answers involve k-anonymity, differential privacy, and membership inference attacks.

Re-identification: the naive approach that never worked

The first instinct is to remove names and account numbers and call the result "anonymized". Sweeney (2000) demolished this instinct in a paper now taught in every privacy course: 87 % of the US population is uniquely identified by the triple (five-digit ZIP code, date of birth, gender). None of those three fields is a "direct identifier" in the ordinary sense. Combined with a public voter roll, this triple lets you name a de-identified medical record. The lesson generalizes: quasi-identifiers — features that individually seem harmless but jointly single out people — do the work of names, and there are far more of them than one thinks.

On the credit dataset, the quasi-identifiers are more than three. Combining ZIP code, employer name, income to the nearest thousand, and month of birth almost surely identifies each applicant uniquely in a city of one hundred thousand people. Removing "PII" without controlling for quasi-identifiers is theatre.

k-anonymity and its refinements

k-anonymity offers a formal defense. A dataset is k-anonymous when, for every combination of quasi-identifier values, at least $k$ rows share that combination. If $k = 5$, no attacker can narrow a match below five candidates using the quasi-identifiers.

Two operations produce k-anonymity: generalization (replace precise values with ranges — 34 years old becomes 30–39, ZIP 75011 becomes 750xx) and suppression (drop the rows that would otherwise stand alone). Both cost accuracy on downstream models, and the cost grows with $k$.

k-anonymity has known weaknesses. If all five people sharing a quasi-identifier combination happen to have the same sensitive value (all defaulted, or all have the same disease), the attacker learns the sensitive value without needing to know which person is which. l-diversity and t-closeness patch this by requiring sensitivity-value diversity within each equivalence class. In practice, most modern releases have moved on to differential privacy.

Differential privacy and the epsilon budget

Differential privacy (Dwork, 2006) shifts the target. Rather than trying to prevent re-identification directly, it demands that the presence or absence of any single individual in the dataset barely changes any statistic released. Formally, a randomized mechanism $M$ is $\varepsilon$-differentially private if, for any two datasets $D$ and $D'$ differing in one row and any set of outputs $S$: $P(M(D) \in S) \leq e^{\varepsilon} \cdot P(M(D') \in S)$.

The parameter $\varepsilon$ is a privacy budget. Small $\varepsilon$ (say 0.1) means strong privacy and heavy noise; large $\varepsilon$ (say 5) means weak privacy and light noise. The choice is a trade-off between utility and protection, and it is cumulative: every query on the dataset consumes budget, and once spent, the budget is spent for that individual, forever.

The Laplace mechanism achieves $\varepsilon$-differential privacy for a numeric statistic by adding Laplace noise scaled to the statistic's sensitivity (the maximum change one row can cause) divided by $\varepsilon$. For counts, sensitivity is 1; for averages, sensitivity is the range of the variable divided by the sample size. Modern libraries — opendp, pydp, TensorFlow Privacy, PyTorch Opacus — handle the accounting so that composition of many queries stays within a chosen total budget.

Applied to model training, differential privacy takes the form of DP-SGD: gradients are clipped per-sample, then Gaussian noise is added at each step. The Opacus wrapper is one line around a PyTorch training loop, and the price is roughly one to three points of accuracy for a moderate budget on the credit scorer.

Membership inference: the model itself as a leak

Even without releasing the dataset, the trained model can leak. Membership inference attacks (Shokri et al., 2017) ask: given a candidate record, was it in the training set or not? The attacker exploits the fact that models tend to be more confident on samples they saw during training than on unseen samples of the same distribution. A logistic regression on the model's output confidence, trained on shadow models, is often enough to reach 70 % or more accuracy on the membership question — a serious problem when the training set is a list of applicants for a sensitive product.

Defenses include: training with differential privacy (which bounds the attack's success by construction), reducing overfitting (a model that generalizes well leaks less), and outputting a smoothed or truncated confidence. The audit records whether membership inference has been tested and what the attack's advantage is.

The privacy chapter of the credit audit

For the credit scorer, the audit records the following. The raw dataset is never shared outside the bank. Any analytical extract used for reporting is k-anonymized with $k = 10$ on the quasi-identifiers (ZIP prefix, age band, gender, employment category). The trained model itself is not released to third parties; if it were, a differential-privacy training with $\varepsilon = 4$ would be applied first. A membership inference attack was run in-house and reached 0.53 accuracy, close to the 0.50 chance level, which is documented in the model card. The next model version will experiment with DP-SGD to bring that number closer to chance.

Summary

• Removing names is not anonymization: quasi-identifiers (ZIP + birth date + gender) uniquely identify most people in a public roll.
• k-anonymity requires each quasi-identifier combination to be shared by at least $k$ rows; l-diversity and t-closeness patch its known holes.
• Differential privacy with an $\varepsilon$ budget guarantees that any single person's contribution barely changes released statistics or trained models.
• Membership inference attacks show that the trained model itself leaks; DP-SGD and less overfitting reduce the leak.

Next module: writing the model card and datasheet that record every choice made across modules 1 to 5.