Module 4 — GAN: generator versus discriminator

The VAE of module 3 works, but it blurs. In 2014 Goodfellow proposed a completely different approach: drop likelihood entirely and let two networks fight. One generates fakes, the other tells fakes from real, and each pushes the other to improve. That is a generative adversarial network, GAN. This module builds a small DCGAN on MNIST, and — just as importantly — teaches you to distrust its loss curves.

The game

Two networks play against each other:

• The generator $G$ takes a noise vector $z \sim \mathcal{N}(0, I)$ and outputs a candidate image $G(z)$.
• The discriminator $D$ takes an image (real from the training set or fake from $G$) and outputs a probability that it is real, in $[0, 1]$.

The discriminator is trained as an ordinary binary classifier with cross entropy: label $1$ for reals, label $0$ for fakes. The generator is trained to make the discriminator get it wrong: fakes should be classified as real.

In the original formulation of Goodfellow, the two objectives are:

$\max_D \; \mathbb{E}_{x \sim p_{\text{data}}} [\log D(x)] + \mathbb{E}_{z} [\log(1 - D(G(z)))]$

$\min_G \; \mathbb{E}_{z} [\log(1 - D(G(z)))]$

This is a minimax game. At its theoretical equilibrium, the generator's distribution matches the data distribution and the discriminator can only guess with probability $\tfrac{1}{2}$.

Non-saturating trick and BCE losses in practice

Nobody trains a GAN with the original generator loss. When $D$ is confident that a fake is fake, $\log(1 - D(G(z)))$ saturates and its gradient vanishes — the generator learns nothing exactly when it most needs to. The non-saturating replacement is:

$\min_G \; -\mathbb{E}_{z} [\log D(G(z))]$

In code, both losses become plain binary cross entropy on $D$'s output:

import torch
import torch.nn as nn
import torch.nn.functional as F

def d_loss_fn(d_real, d_fake):
    real = F.binary_cross_entropy_with_logits(d_real, torch.ones_like(d_real))
    fake = F.binary_cross_entropy_with_logits(d_fake, torch.zeros_like(d_fake))
    return real + fake

def g_loss_fn(d_fake):
    return F.binary_cross_entropy_with_logits(d_fake, torch.ones_like(d_fake))

Note that we use _with_logits: the discriminator outputs a raw scalar, and applying the sigmoid inside the BCE is numerically safer.

DCGAN on MNIST

DCGAN — deep convolutional GAN, Radford 2015 — is the small architecture that made GANs actually work on images. Its recipe is short: transposed convolutions in $G$, strided convolutions in $D$, batch normalisation everywhere except the first layer of $G$ and the last of $D$, LeakyReLU in $D$, ReLU in $G$, and Tanh at the output.

class Generator(nn.Module):
    def __init__(self, z_dim=100, n_channels=1):
        super().__init__()
        self.net = nn.Sequential(
            nn.ConvTranspose2d(z_dim, 128, 7, 1, 0), nn.BatchNorm2d(128), nn.ReLU(True),
            nn.ConvTranspose2d(128, 64, 4, 2, 1), nn.BatchNorm2d(64), nn.ReLU(True),
            nn.ConvTranspose2d(64, n_channels, 4, 2, 1), nn.Tanh(),
        )

    def forward(self, z):
        return self.net(z.view(z.size(0), z.size(1), 1, 1))

class Discriminator(nn.Module):
    def __init__(self, n_channels=1):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(n_channels, 64, 4, 2, 1), nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(64, 128, 4, 2, 1), nn.BatchNorm2d(128), nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(128, 1, 7, 1, 0),
        )

    def forward(self, x):
        return self.net(x).view(-1)

Sizes chosen for MNIST: the generator takes a 100-dimensional noise and returns a $1 \times 28 \times 28$ image; the discriminator maps the same shape back to a single logit.

The alternating training loop

The two networks are optimised in turn, one step each per batch. A common recipe:

device = "cuda" if torch.cuda.is_available() else "cpu"
G = Generator().to(device)
D = Discriminator().to(device)
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))

for epoch in range(30):
    for x, _ in loader:  # from module 2
        x = x.to(device)
        z = torch.randn(x.size(0), 100, device=device)

        # 1) Discriminator step
        with torch.no_grad():
            fake = G(z)
        d_real = D(x)
        d_fake = D(fake)
        loss_d = d_loss_fn(d_real, d_fake)
        opt_d.zero_grad(); loss_d.backward(); opt_d.step()

        # 2) Generator step
        fake = G(z)
        loss_g = g_loss_fn(D(fake))
        opt_g.zero_grad(); loss_g.backward(); opt_g.step()

The betas=(0.5, 0.999) for Adam is the second half of the DCGAN recipe. The default betas=(0.9, 0.999) makes GAN training oscillate; that little change stabilises it. It is not folklore, it is in the paper, and it matters.

Reading GAN loss curves

Here is where GANs get confusing. Plot loss_d and loss_g over training and try to conclude anything from them. You cannot.

At perfect equilibrium, both losses tend to $\log 2 \approx 0.69$. But intermediate values are almost meaningless: loss_d decreasing does not mean anything is improving, since it can decrease when $G$ is producing garbage that $D$ classifies trivially. loss_g decreasing does not mean fakes are more realistic, since it can decrease when $D$ has become weak.

A steady GAN loss curve is not a sign of good training

Unlike classification, GAN losses are not a scalar measure of quality. Two loss curves at $0.7$ can hide a healthy generator or a broken one; two loss curves oscillating wildly can hide a healthy generator or a broken one. The only real diagnostic is to generate a fixed set of samples periodically — say every epoch, from the same noise vector — and look at them.

Fix a batch of noise once, save it, and generate from it after every epoch:

fixed_z = torch.randn(64, 100, device=device)  # created once, before training

# inside the epoch loop:
with torch.no_grad():
    samples = G(fixed_z).cpu()
# save or display samples

The evolution of those images is the diagnostic. Loss curves are supplementary information, useful mostly to detect the pathologies of module 5.

The two rules that make DCGAN work

Two invisible details break DCGAN training when ignored.

Label smoothing on the real side. Rather than using label $1.0$ for reals, use $0.9$. This is called one-sided label smoothing. It prevents the discriminator from becoming absolutely confident, which in turn prevents its logits from saturating and killing the generator's gradient.

Batch normalisation, but not everywhere. The output layer of the generator and the input layer of the discriminator go without batch norm. Putting batch norm at the generator's output pushes it to produce a fixed statistical footprint that betrays fakes; putting it at the discriminator's input smooths over the exact input statistics we want it to detect. Both are counterintuitive if you learnt batch norm as "always beneficial", but both are documented.

What the model produces on MNIST

After 30 epochs on MNIST, a fresh DCGAN produces digits that are visibly sharper than the VAE of module 3. Strokes are crisp, characters have decisive shapes. But look at 64 samples from the same fixed noise batch and you may find that only six or seven digit classes appear — the network has quietly stopped generating the other three or four. That is mode collapse, and it is the main subject of module 5.

In summary

• A GAN is a two-player game: the discriminator classifies reals against fakes, the generator tries to fool it. Losses are cross-entropies computed with logits.
• The non-saturating generator loss $-\log D(G(z))$ replaces the theoretical $\log(1 - D(G(z)))$ because the latter's gradient vanishes when $D$ is confident.
• DCGAN's recipe is not folklore: Adam with betas=(0.5, 0.999), batch norm except at $G$'s output and $D$'s input, LeakyReLU in $D$, one-sided label smoothing.
• GAN loss curves are almost useless as a quality signal; monitor a fixed set of samples across epochs to see whether the generator is actually improving or collapsing.

Next module: mode collapse, the pathology hinted at above, and the Wasserstein / spectral / gradient-penalty tricks that keep GAN training alive.