Module 1 — Discriminative and generative models

Every model you have written so far in the parcours answered the same question: given an input, predict a label. A classifier picks a class, a regressor picks a number. That family is called discriminative. This course is about a different family, one that is harder to train and harder to evaluate — but that draws new examples that look like the training set.

Two questions, two conditional probabilities

A discriminative model estimates the probability of a label given the input:

$p(y \mid x)$

Given a photograph $x$, what is the probability that it shows a cat rather than a dog? The model does not have to know what a cat looks like in absolute terms; it only has to know which of the ten thousand features it extracts separates the two labels.

A generative model estimates the probability of the input itself:

$p(x)$

or, when we also want conditioning, the joint distribution $p(x, y)$. Given a class "cat", what probability does the model assign to every possible image? The answer must be higher for a photograph of a cat than for uniform noise, higher for a plausible cat than for a cat with three eyes.

The difference is not academic. To classify, the model can ignore whatever does not help separate the classes — the background, the lighting, the pose. To generate, it must model the whole world of images, because a sampled image will contain a background, some lighting and a pose whether the model likes it or not.

Sampling is what makes a model generative

The mark of a generative model is that you can draw from it. Given a trained model of $p(x)$, running the sampling procedure produces a new $x$ that ideally looks like it came from the training set but is not one of its members.

We will build three sampling procedures in this course, each with its own trade-offs:

Family	How it samples	Speed	Quality	Diversity
Variational autoencoder	draw $z \sim \mathcal{N}(0, I)$, decode $x = g(z)$	very fast	blurry	good
GAN	draw $z \sim \mathcal{N}(0, I)$, pass through generator	very fast	sharp	often narrow
Diffusion	start from noise, denoise for $T$ steps	slow	very sharp	good

A discriminative classifier has no sampling procedure. Asking it to "generate a cat" makes no sense: it only ranks inputs against a label.

Likelihood: what the training signal looks like

Discriminative training minimises the negative log likelihood of the label. Generative training tries to minimise the negative log likelihood of the data itself, $-\log p(x)$, but that quantity is often intractable to compute for models with a latent variable.

Three responses have shaped the field. The variational autoencoder maximises a lower bound on $\log p(x)$, the ELBO. The GAN abandons likelihood entirely and replaces it with an adversarial game against a discriminator. Diffusion models train on a simplified regression loss that is provably equivalent to maximising a variational bound, but that looks nothing like a likelihood at first sight.

That three families exist is not a redundancy of the field: each solved a different obstacle in the same original problem.

"High likelihood" does not mean "looks real"

It is possible to build a generative model that assigns very high likelihood to a training set and produces awful samples. Autoregressive models over pixels are the classic example. Conversely, GANs assign undefined likelihood to their samples yet produce photorealistic images. Log likelihood, sample quality and mode coverage are three different axes — module 8 comes back to this.

The rise, retreat and return of each family

Understanding the current state of the field is easier with a rough chronology.

Autoencoders in their simplest form date back to the 1980s but were not used for generation — they compressed. Variational autoencoders (Kingma and Welling, 2013) turned them into generators by imposing a well-behaved latent distribution. Their samples were immediately blurry, but the framework was clean and the training stable.

Generative adversarial networks (Goodfellow et al., 2014) took the field by surprise. The samples were sharp, the framework was intuitive, and every conference of 2015 to 2019 was dominated by new variants. Then a stubborn problem — training instability and mode collapse — capped their real usefulness for open-ended generation.

Denoising diffusion probabilistic models (Ho et al., 2020) started as a curiosity. Within two years they had displaced GANs on image benchmarks, powered Stable Diffusion, and become the de facto choice for image, audio and even protein generation. They are slow to sample from, but every other trade-off is favourable.

Which family for which problem

A rough guide, revisited in the recap:

• Sharp faces or objects, unconditional or lightly conditional: diffusion first, GAN if latency matters more than quality.
• A smooth latent space to interpolate through and reason about: VAE, sometimes a variant with a discrete latent.
• Text: neither of the three above. Autoregressive transformers (module 9 of course 12) still dominate, though discrete diffusion is an active research direction.
• Audio: diffusion or autoregressive, depending on whether latency or quality wins.

The MNIST digits we will generate in modules 2 to 5 are the smallest dataset on which the three families show visibly different behaviour. That is why we start there rather than with faces.

The tools we will need

The whole course runs in PyTorch. A minimal setup that will be reused across every module:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

device = "cuda" if torch.cuda.is_available() else "cpu"

transform = transforms.Compose([
    transforms.ToTensor(),                 # values in [0, 1]
    transforms.Normalize((0.5,), (0.5,)),  # values in [-1, 1]
])

mnist_train = datasets.MNIST(root="./data", train=True, download=True, transform=transform)
loader = DataLoader(mnist_train, batch_size=128, shuffle=True, num_workers=0)

x, y = next(iter(loader))
print(x.shape, x.min().item(), x.max().item())  # (128, 1, 28, 28) -1.0 1.0

We normalise MNIST to $[-1, 1]$ from the start because both the DCGAN of module 4 and the diffusion model of module 6 expect that range. Every subsequent module will import this snippet or a variant of it.

In summary

• A discriminative model estimates $p(y \mid x)$ and ranks inputs against a label; a generative model estimates $p(x)$ and can be sampled from to produce new inputs.
• Three families now dominate: VAE (stable but blurry), GAN (sharp but hard to train, module 5) and diffusion (sharp and stable but slow to sample from, module 6).
• Likelihood is often intractable for latent variable models; each family sidesteps this differently — variational bound, adversarial game, or denoising regression.
• High log likelihood and realistic samples are not the same axis; module 8 introduces FID precisely because likelihood alone does not judge a generator.

Next module: autoencoders and the latent space they carve out — the object that variational autoencoders will turn into a generator.