#gan — Deep learning

GAN: a forger against an inspector

What you'll play with

Welcome to #gan. On screen, a plane seen from above: the white points are the originals (here a ring), the pink points are the fakes produced by a small network G, the generator — anywhere for now. The grey background is the map of a second network D, the discriminator: it hesitates everywhere (50%). The game: G is a forger that never sees the originals, only D's verdict; D is an inspector that sees reals and fakes and learns to tell them apart. Each improves against the other, until the inspector can no longer decide — that is the GAN principle, which produced the first photorealistic synthetic faces.
Run the first twenty rounds: /step 20. Each round, D sees 16 reals and 16 fakes and tunes its criteria; then G receives D's verdict on its fakes, backpropagates the gradient through D down to its own weights, and adjusts.
Keep going: /step 100. The dashed lines on the panel mark the equilibrium levels: ln 2 ≈ 0.69 for G and 2·ln 2 ≈ 1.39 for D — what you get when D answers 50% everywhere. A very low D loss means the inspector is winning; a rising G loss means the forger is getting caught.
One more /step 100. Watch the right-hand preview: D's accuracy on the reals and on the fakes. When both hover around 50%, the inspector does no better than a coin flip — exactly what we want.
Change the originals: /real two-modes. The forger keeps its ring reflexes and the inspector its criteria: nothing is reset.
Here is the most famous GAN pitfall. Type /collapse: the forger will play 16 moves at the maximum learning rate for a single, tiny move by the inspector. In other words, G is optimised against a nearly frozen D.
Restart training: /step 60.
Your turn: /collapse to switch the regime off, /real spiral then /step 100 (a shape that is harder to imitate), /noise 1 (a one-dimensional noise: G can only draw a curve), /lr-d 0.1 (an inspector that is too fast: D becomes overconfident and G's gradient vanishes), /sample 300 to see more fakes, /seed 12 for a different game, /reset to start over. Next step: the #tokenization channel, the first channel of the LLM theme — how a language model splits text before even learning it.

Channel commands

/real <circle|two-modes|spiral|square> — Change the real distribution to imitate (G and D kept).
/step <1..100> — Alternating training iterations: 2 D steps then 1 G step, batches of 16.
/noise <1..4> — Latent noise dimension z (resets G, D kept).
/lr-g <0.001..0.1> — Generator learning rate.
/lr-d <0.001..0.1> — Discriminator learning rate.
/sample <50..300> — Number of fakes displayed (same noise z, extended or truncated).
/seed <1..9999> — New game: other reals, other initial weights, same settings.
/collapse — Arm (or switch off) the regime that triggers mode collapse: maximum lrG, tiny lrD, 16 G steps per D step.
/reset — Back to the ring, d = 2, lrG = lrD = 0.02, 200 fakes, fresh networks.

Glossary

GAN (generative adversarial network): A pair of networks trained against each other: a generator fabricates fakes from noise, a discriminator learns to tell them from the reals. The generator never sees the data, only the discriminator's verdict — and yet it ends up imitating their distribution.
Generator: The network G that turns a noise vector z into a sample (here a 2D point, elsewhere an image). It is trained to maximise the probability that D is wrong: its gradient flows back through D down to its own weights.
Discriminator: A binary classifier D that receives a sample and estimates the probability it is real. It is trained on reals (target 1) and fakes from G (target 0). Its probability map also serves as an "error signal" for the generator.
Zero-sum game / minimax: GAN formulation: D maximises ln D(x) + ln(1 − D(G(z))), G minimises the same quantity. What one gains, the other loses. In practice G optimises −ln D(G(z)) instead, called the non-saturating loss, which gives a useful gradient even when D dominates.
Latent noise: A random vector z (Gaussian, dimension d) that feeds the generator: it is the only source of diversity for the fakes. Two nearby z give two nearby fakes; the dimension of z bounds the "shape" G can produce (a curve if d = 1, a surface if d ≥ 2).
Mode collapse: A classic failure mode where G only produces one or a few samples (a single "mode") for all z, because it has found the place D rates highest. G's loss can still look fine: it does not measure diversity. Here we spot it by the fakes' standard deviation collapsing.
Nash equilibrium: A situation where neither player has any interest in changing strategy alone. For an ideal GAN: G reproduces the reals' distribution exactly and D answers 50% everywhere; the losses are then ln 2 for G and 2·ln 2 for D. In practice you only orbit around it.
Alternating training: You cannot optimise G and D at the same time: each iteration you take a few D steps (G frozen), then one G step (D frozen), on small batches. The ratio between the two speeds (learning rates, step counts) decides who dominates — and thus the game's stability.
Vanishing generator gradients: When D becomes too good, its sigmoid saturates: D(G(z)) ≈ 0 everywhere and the derivative flowing back to G tends to zero. The forger stops learning while it is still bad. The non-saturating loss and a not-too-fast D (moderate lr-d) limit the problem.
Conditional GAN: A variant where G and D also receive a context (a class label, some text, an image): G then learns to produce a sample of that category. This is the basis of guided generators (image from a sketch, from a word…).

Other channels in Deep learning

#optimizers — SGD, Momentum and Adam: the race to the minimum.
#batch-normalization — Batch normalization: keeping activations in the right range.
#rnn-lstm — RNN and LSTM: remembering a sequence.
#autoencoder — Autoencoder: compress then reconstruct.
#transfer-learning — Transfer learning: start from an already-trained network.
#gan — GAN: a forger against an inspector