Lesson 4 — Privacy, consent and data
Privacy in machine learning is not mainly about storing data securely. It is about four things that traditional privacy thinking handles badly.
1. Consent does not survive contact with scale
The consent model assumes you can tell someone what their data will be used for. Machine learning breaks that assumption in three ways.
Purpose is not known in advance. Data collected to operate a service is later used to train a model for something nobody had thought of. Consent given for the first does not cover the second in any meaningful sense.
Web-scale collection has no consent at all. Foundation models are trained on scrapes of the public internet: personal blogs, forum posts, photographs, code, medical questions asked in public. Nobody consented, most did not know it was possible, and "it was publicly accessible" is a claim about access rather than about permission.
Withdrawal is not implementable. Deleting a record from a database is straightforward. Removing its influence from a trained model is not — the practical answer is retraining, which is expensive and rarely done. Research on machine unlearning is active and not yet a general solution, which means a right to erasure is, for models, largely unenforced.
The honest summary: for large models, consent as ordinarily understood does not apply, and pretending otherwise is worse than acknowledging the gap.
2. Models leak their training data
A model is not a summary that discards the specifics. It can memorise, particularly content that appeared repeatedly.
Demonstrated results: verbatim passages extracted from language models, near-identical training images extracted from diffusion models, and personal details reproduced when prompted appropriately. Sequences appearing many times in training are the most vulnerable, which includes duplicated documents and personal data that was scraped from multiple sources.
Two attack classes matter here:
Membership inference determines whether a specific record was in the training set. Consequential when membership is itself the sensitive fact — that someone participated in a study of a particular condition, for instance.
Model extraction approximates a model by querying its API repeatedly, which is a commercial rather than a privacy concern but shares the mitigations.
Practical implications: if you fine-tune on data containing personal information, treat the resulting model artefact as containing personal information — with the access controls, retention rules and deletion obligations that follow. Deduplicate training data, since duplication drives memorisation. And consider differential privacy where the data warrants it, accepting an accuracy cost in exchange for a mathematical bound on what any individual record contributes.
3. Inference reveals what was never collected
The most underappreciated point. Models infer sensitive attributes from innocuous data, so not collecting something does not prevent knowing it.
Documented inferences from ordinary behavioural data include pregnancy from purchasing patterns, sexual orientation from social connections, mental health state from language use and posting rhythm, and political alignment from unrelated preferences. None of these attributes was collected.
Two consequences that data protection frameworks handle awkwardly:
Not collecting is not protection. A system that never asks about health can still act on inferred health status.
Group inference affects people who never interacted with you. A model that learns a pattern about a demographic applies it to individuals who never supplied any data at all.
This is why "we don't collect sensitive data" is an incomplete answer to a privacy question. The right question is what the system can infer and act on.
4. Anonymisation promises more than it delivers
Removing names and identifiers is necessary and insufficient.
Re-identification results are consistent across decades: a small number of quasi-identifiers — postcode, date of birth, sex — suffice to single out most individuals in a population. Anonymised medical, mobility, streaming and search datasets have all been re-identified by linking against public information.
| Approach | What it actually gives |
|---|---|
| Removing names | Very little on its own |
| Removing all direct identifiers | Still re-identifiable through combinations |
| Generalising values into buckets | Better, and reduces utility |
| k-anonymity | Each record indistinguishable from k−1 others; vulnerable to linkage |
| Differential privacy | A mathematical bound on individual contribution; costs accuracy |
| Synthetic data | Useful, and can still leak if the generator memorised |
The realistic position: treat anonymisation as risk reduction rather than elimination, and keep the access controls and purpose limitation you would apply to identifiable data.
What the law asks, in outline
Not legal advice, and the recurring themes are worth knowing since they shape what you will be asked.
A lawful basis for processing, of which consent is only one and often not the most appropriate.
Purpose limitation. Data collected for one purpose is not automatically available for another, and "training a model" is a distinct purpose requiring its own basis.
Data minimisation. Collect what the purpose needs. This sits in genuine tension with a field whose instinct is to gather everything in case it helps.
Rights of access, correction and erasure, which as noted are technically awkward once data is inside a trained model.
Special protection for sensitive categories — health, ethnicity, beliefs, biometrics, sexuality — where the threshold for processing is markedly higher, and where inference of these attributes raises questions that frameworks are still working out.
Rules on solely automated decisions with significant effects, including in several frameworks a right to human involvement and to meaningful information about the logic.
Cross-border transfer restrictions, which is where lesson 4 of the cloud course becomes an architecture constraint.
A practical baseline
- Document your lawful basis for training, separately from your basis for operating the service.
- Minimise deliberately. Every field you keep is a field you must protect and justify.
- Deduplicate training data, because duplication drives memorisation.
- Classify the model artefact according to what its training data contained.
- Test for memorisation on anything fine-tuned on personal data — prompt it for the specifics and see what comes out.
- Ask what the system can infer, not only what it stores.
- Consider differential privacy where the data is sensitive and the accuracy cost is acceptable.
Stop asking "is this data protected?" and ask "what could this system reveal or infer about a person, and would they consider that acceptable?" That question catches inference and memorisation risks, which the storage-focused framing misses entirely.
In three sentences
Consent does not survive machine learning at scale: purposes are unknown at collection time, web-scale training involves no consent at all, and withdrawal is not implementable because removing a record's influence from a trained model effectively means retraining. Models also leak — memorising duplicated content and reproducing it verbatim, and exposing whether a specific record was in the training set — which means a model fine-tuned on personal data should be classified and controlled as containing personal data. The two points that storage-focused privacy thinking misses entirely are that inference reveals attributes never collected, so not collecting something does not prevent acting on it, and that anonymisation is risk reduction rather than elimination, since a handful of quasi-identifiers suffices to re-identify most individuals.