Module 9 — Generating text, images and audio

Modules 2 to 8 focused on image generation, because that is where the three families are visually easiest to compare. Real applications rarely stop there. This module widens the frame: what generates text, what generates audio, why the winning architecture is not the same across modalities, and how the cost per sample varies by three orders of magnitude between them.

Two dominant families across modalities

Every generative model in production today falls into one of two categories, with occasional hybrids.

Autoregressive models generate one token at a time, conditioning each new token on all previous ones:

$p(x) = \prod_{i=1}^{n} p(x_i \mid x_{<i})$

The order in which tokens are consumed matters: left-to-right for text, raster for images, first-to-last sample for audio. Transformers dominate this family since 2017. Training is a plain likelihood maximisation with cross-entropy on the next-token distribution.

Diffusion models — the ones from modules 6 and 7 — start from noise and denoise iteratively, applying a global transformation at each step rather than adding a token at a time. Autoregressive models are strictly sequential at sampling time; diffusion models are sequential in the number of denoising steps but parallel across positions within a step.

Their strengths differ enough that no modality picks one uniformly.

Text: autoregressive wins

For text, autoregressive transformers are the near-exclusive winner in 2026: GPT, Llama, Claude, Gemini, Mistral, Qwen — all of them are autoregressive over subword tokens, all trained with next-token prediction and reinforcement fine-tuning from human preferences.

Two reasons make autoregression natural for text. First, text has a strong linear order: characters follow characters, words follow words, and the whole reading experience is sequential. Predicting the next token given all previous ones matches how humans consume text. Second, the vocabulary is discrete: a subword tokeniser produces roughly 50,000 to 200,000 possible tokens, and softmax over that vocabulary is a familiar objective.

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2").eval()

prompt = "Once upon a time"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, do_sample=True, top_p=0.9, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Discrete diffusion for text is an active research direction (SEDD, MDLM, D3PM), but no discrete-diffusion text model at the frontier level ships in production as of early 2026. This may change; it has not yet.

Images: diffusion has taken over

Modules 2 to 7 already showed why. On images, diffusion produces sharper samples than a VAE, is much more stable than a GAN, and conditions cleanly on text through cross-attention. All the top open-weight image generators of 2026 — Stable Diffusion 3, Flux, Playground — are diffusion models over latents.

Autoregressive image models still exist (Parti, VAR), and they excel when the goal is very high compositional fidelity to a complex prompt, because the sequential process lets each new region be aware of what has already been placed. But they are slower to sample from and harder to control with guidance, so diffusion currently dominates.

Audio: the split by task

Audio is the most fragmented modality, because the tasks are so different that they call for different families.

Text-to-speech. Modern systems typically use a diffusion model on spectrograms followed by a vocoder that turns the spectrogram back into a waveform. Bark, XTTS and Voicebox are of this family. Autoregressive speech tokenisers like AudioLM followed by non-autoregressive generation are the other successful design.

Music generation. Diffusion on latent representations of audio produces the current state of the art (Stable Audio, MusicLM). Long-form musical coherence — a five-minute piece that develops a theme — remains an open problem, and hybrid designs that combine an autoregressive planner with a diffusion synthesiser are an active research direction.

General audio and sound effects. Diffusion again, on similar latent representations. The training data is much smaller than for text or images, so open weights lag behind proprietary systems.

from transformers import pipeline

speech = pipeline("text-to-speech", model="suno/bark-small")
audio = speech("Welcome to the generative audio module.")
# audio["audio"] contains the waveform samples, audio["sampling_rate"] the rate

The cost of generation, by modality

The wall-clock cost of one sample varies by three orders of magnitude, and it matters for anyone shipping a generative product.

Modality	Model example	Cost per sample	Latency
Text, one paragraph	GPT-4 class, 200 tokens	fractions of a cent	1 to 5 s
Text, long form	GPT-4 class, 5000 tokens	a few cents	30 s to minutes
Image, 1024 by 1024	Stable Diffusion 3, 30 steps	1 to 5 cents on rented GPU	3 to 8 s
Audio, 10 seconds of speech	Bark class	around 1 cent	5 to 10 s
Video, 5 seconds at 720p	Sora class	20 cents to 1 dollar	30 s to a few minutes

The costs shift downwards month over month, but the ratios between modalities remain roughly stable. When designing a product, cost-per-sample often dictates architecture: a real-time voice avatar cannot afford to run a large speech diffusion model in the loop, so it uses a smaller autoregressive model with a lightweight vocoder.

Cost estimates are not linear in output length

For autoregressive models the cost scales roughly linearly in output length in tokens, but with a constant compute per token that grows with the context length because attention is quadratic. For diffusion models the cost is roughly constant in output length within a fixed resolution — 30 steps is 30 steps whether the image is empty or crowded — but scales super-linearly in resolution because each step processes more pixels. Napkin math for one modality does not transfer to another.

Multimodality: the arrival that changes both markets

Since 2023, the frontier models are multimodal: they consume and produce a mixture of text, images and audio. GPT-4o, Gemini 2, Claude with Vision — each can look at an image, hear a voice, read a document, and reply in text plus optionally a synthesised voice.

Architecturally, multimodality is currently achieved by attaching modality-specific encoders to a shared transformer backbone. The image encoder is often a Vision Transformer or a variant, the audio encoder a spectrogram encoder or a codec, and the outputs are projected into the transformer's token space so that images and audio become sequences of "soft tokens" alongside text tokens.

Generating out of the model in multiple modalities is harder. The frontier approach in early 2026 is to produce text tokens autoregressively and then hand off to specialised diffusion generators (image, audio) at the model's direction. Fully unified generation — a single model that autoregressively predicts pixels, audio samples and text tokens — is an active research goal, not yet a shipped product.

Choose the modality your data has, not the one that is fashionable

It is tempting to add image generation to a product because the current wave of demos does. If the underlying data is text — support tickets, contracts, code — a good text model will produce more value at lower cost than an image generator wedged onto the side. The winning generative product is usually the one whose main modality matches the user's actual task.

In summary

• Autoregressive transformers dominate text generation; diffusion models dominate images; audio is split by task between the two families, often with hybrid pipelines.
• The choice of family is not a preference: text's linear order and discrete vocabulary favour autoregression, images' continuous pixel values and parallel structure favour diffusion, audio depends on the task.
• Costs per sample vary by three orders of magnitude across modalities, and by resolution or length within one modality; the trade-off drives real product decisions.
• Multimodal frontier models currently attach encoders to a shared transformer and delegate rich-media output to specialised generators; unified end-to-end generation is not yet a shipped product.

Next module: the questions that generation raises about training data, copyright, consent, provenance and hyperrealistic fakes.