Skip to main content

Lesson 5 — The limits that matter

These are not caveats appended for balance. Each one has produced real consequences for real organisations, and each is a property of how the technology works rather than a temporary shortfall.

1. Fabrication is structural​

Lesson 3 established the mechanism: a correct citation and an invented one are produced identically, because the model constructs plausible sequences and has no separate faculty for knowing.

The consequences have been documented repeatedly. Lawyers have submitted filings citing cases that do not exist, and been sanctioned for it. Support chatbots have invented refund policies their operators were then held to. Published articles have contained fabricated quotations attributed to real people.

The pattern in every case is the same: the output was fluent, internally consistent and plausible, so nobody checked. Fluency is what makes fabrication dangerous, not the error rate.

What reduces it:

  • Supply the source material and instruct the model to answer only from it.
  • Require citations and check them, understanding that the citation itself may be invented.
  • Prefer extractive over generative output wherever the choice exists.
  • Use generation to draft and a human or a test to verify.
  • Design for the model to decline. A system that must always answer will always answer, including when it should not.

What does not reduce it: a larger model, a lower temperature, or asking the model whether it is sure.

Three separate questions get conflated, and they have different answers.

Was training on copyrighted material lawful? Unsettled and actively litigated in multiple jurisdictions. Arguments for fair use or text-and-data-mining exceptions face arguments that commercial models built on unlicensed work are straightforward infringement. Rulings so far have gone in different directions on different facts. Anyone stating this confidently in either direction is overreaching.

Can generated output be copyrighted? In several jurisdictions, including the United States, purely machine-generated work has been held not to attract copyright, on the basis that authorship requires a human author. Substantial human creative contribution can change that for the human-contributed parts. The practical implication is frequently overlooked: you may be unable to prevent anyone else from using output you generated.

Can output infringe someone's copyright? Yes, in specific circumstances. Which brings us to memorisation.

3. Memorisation and leakage​

The general claim that a model stores patterns rather than copies is true. The exception matters.

Content appearing many times in training data can be reproduced closely. Researchers have extracted near-identical copies of training images from diffusion models and verbatim passages from language models. The effect concentrates on duplicated content — stock photographs, famous artworks, widely-quoted text, and, importantly, personal data that appeared repeatedly in scraped sources.

Two practical implications:

Output resembling an existing work is a real possibility, particularly when you prompt with an artist's name or a recognisable property. Reverse image search on anything you intend to publish commercially costs nothing.

Anything you put into a hosted model may be retained. Check whether inputs are used for training, whether they are logged, and for how long. Organisations have leaked internal source code and confidential documents this way. Where the answer is unacceptable, locally-run models are the reason that ecosystem exists.

Voice cloning now works from seconds of sample audio. Video generation is short but improving quickly. Face swapping is available in consumer applications.

The documented harms, in rough order of measured prevalence:

Non-consensual intimate imagery, overwhelmingly targeting women, which is by a wide margin the most common malicious application and the one that receives the least attention relative to its scale.

Fraud. Cloned voices used to authorise transfers, including a case involving a video call with multiple synthetic participants and a loss in the tens of millions.

Political disinformation, though the more corrosive effect may be indirect: once anything could be fake, genuine evidence becomes deniable.

Harassment and reputational attack at individual scale, where the victim has little recourse.

Detection is not a solution. Detectors are unreliable and their false positives cause harm of their own. Provenance — cryptographically signing content at capture and preserving that signature — is more promising, and requires adoption across the whole chain to matter.

5. Bias, inherited and amplified​

Generative models reproduce the statistical patterns of their training data, including who appears in what role.

Documented findings: image models asked for "a doctor" or "a CEO" have returned predominantly male, predominantly light-skinned results at rates exceeding real-world demographics; requests for "a criminal" have skewed towards darker-skinned faces; "beautiful" defaults towards a narrow and specific standard. Language models produce different-quality output across dialects and reproduce occupational stereotypes.

Amplification is the part that gets missed. The model does not merely reflect its data's skew — it tends to sharpen it, because generation concentrates probability on the most likely output. And since generated content increasingly becomes training data for the next generation, the skew compounds. This connects to model collapse: models trained substantially on synthetic output lose diversity and drift towards the average of the average.

Mitigations exist — prompt specificity, filtered training data, output diversity measures — and none makes the problem go away.

6. Cost, energy and water​

Training a large model consumes energy at industrial scale. The figure that gets less attention is inference: a widely-used model serves billions of requests, and the cumulative cost of serving exceeds the one-off cost of training by a growing margin. Data centre cooling also consumes water, which matters where they are sited in water-stressed regions.

The engineering response is genuine and partial: smaller distilled models, quantisation, caching, and simply not calling a large model for tasks a small one handles. The practical version for you is that choosing the smallest model that works is simultaneously the cheapest, fastest and least wasteful decision, which is a rare alignment worth exploiting.

7. Labour and disclosure​

Two things worth stating plainly.

Systems are made safe partly through human annotation of harmful content, work that has been outsourced to low-wage contractors under conditions documented as psychologically damaging. The polished product depends on it.

And a growing set of contexts now expect disclosure of generated content: journalism, academic submission, some advertising regulation, some employment processes. Norms are moving faster than law. Assuming disclosure is expected is the safer default.

In three sentences​

Fabrication is structural rather than a bug, and fluency is what makes it dangerous — lawyers have been sanctioned for citing invented cases and companies held to policies their chatbots made up — so anything factual requires grounding in supplied sources or independent verification. The copyright position separates into three unsettled questions, of which the most overlooked is that purely machine-generated output may not attract copyright at all, meaning you may be unable to stop others using it, while memorisation of duplicated training content makes accidental resemblance to existing work a genuine risk. Synthetic media harms are concentrated in non-consensual imagery and fraud rather than the political scenarios that get discussed, bias is both inherited and amplified with generated content feeding back into future training, and the honest engineering response to cost is to use the smallest model that does the job.

Next — Lesson 6: recap and FAQ →