#autoencoder — Deep learning

Autoencoder: compress then reconstruct.

What you'll play with

Welcome to #autoencoder. On the left, a cross drawn on 8×8 = 64 pixels; in the middle, a bottleneck of 4 neurons; on the right, what the network reconstructs from those 4 numbers alone. Weights are still random: the output is grey mush. An autoencoder learns to compress (encoder: 64 → 4) and then reconstruct (decoder: 4 → 64) by minimising the gap between input and output. No labels: the target is the input itself. That is the idea behind learned compression, denoising and part of image generators.
Start training: /train 20. At each epoch, the network sees the 12 glyphs, measures the reconstruction error and adjusts its weights by backpropagation.
40 more epochs: /train 40. The mean squared error (MSE) shown under the right grid should melt away.
Let's tighten the bottleneck: /latent 2. The network is reinitialised with only 2 neurons in the middle: each glyph must fit into two numbers, exactly the two axes of the latent plane.
Retrain this narrow network: /train 40. Then compare the mean error across the 12 glyphs (right panel) with the one you got at 4 dimensions.
Change the input: /image circle. The same weights encode and then decode a completely different drawing, without any relearning.
Damage the input: /noise 0.3 replaces 30% of the pixels with random values. The network itself has never seen noise.
Travel through latent space: /interpolate cross circle places 5 intermediate points between the codes of the cross and the circle, then decodes each of them.
Your turn: /point 0.5 -0.5 decodes a free point of the latent plane, /pixel 1 1 toggles a pixel off the pattern to watch the network erase it, /latent 8 then /train 30 for a wide bottleneck, /noise 0.6 for a harder denoising, /seed 3 for another initialisation, /reset to start over. Next up: the #transfer-learning channel, where an already-trained encoder becomes the starting point of a new task.

Channel commands

/latent <k=1..8> — Change the bottleneck dimension and reinitialise the network (fresh weights, epoch 0).
/train <epochs=5..60> — Continue training (SGD, target = clean glyph; noisy input if /noise > 0).
/image <cross|circle|bar|line|diagonal|square|triangle|dot|L|T|checker> — Change the input glyph (the network itself does not change).
/noise <0..1> — Corrupts the input (fraction of pixels replaced by a random value); the target stays clean.
/interpolate <a> <b> — Decode 5 intermediate codes between the latent codes of two glyphs (bottom band).
/point <z1> <z2> — Decode a free point of the latent plane (z1, z2; the other dimensions are 0).
/pixel <row=1..8> <col=1..8> — Toggle an input pixel (row 1 at the top) to test a shape outside the dataset.
/seed <n=1..999> — Change the seed: different weight initialisation and different noise pattern; training restarts from zero.
/reset — Back to the initial state: cross, bottleneck 4, random weights, no noise.

Glossary

Autoencoder: Network trained to reproduce its input after squeezing it through a smaller representation. No labels: the target is the input itself, which makes it self-supervised learning.
Encoder: First half of the network: it compresses the input (64 pixels here) into a latent code of k numbers.
Decoder: Second half of the network: it reconstructs an output the same size as the input from the latent code alone. It is also a small generator: give it a made-up code and it produces an image.
Bottleneck: Narrowest layer, between encoder and decoder. Its size k sets the compression ratio (64 → k) and forces the network to keep only the essentials.
Latent space: k-dimensional space where the codes live. Two similar inputs have nearby codes; moving through it means varying the features the network has learned to detect.
Reconstruction error: Gap between output and input, here the mean squared error (MSE) over the 64 pixels. It is the only loss minimised during training.
Denoising: Variant where a noisy input is presented while the target remains the clean image: the autoencoder learns to remove noise and its representation becomes more robust.
Latent interpolation: Pick intermediate points between two codes and decode them: you get shapes that exist in no dataset. It reveals whether the latent space is continuous or full of holes.
Variational autoencoder: Autoencoder whose code is a distribution (mean and variance) rather than a point, with a penalty that pulls codes around the origin. The latent space becomes smooth and samplable: a true generative model.
Dimensionality reduction: Representing high-dimensional data (64 pixels) with few numbers (k) while losing as little information as possible. The autoencoder is a non-linear version; PCA is the linear one.

Other channels in Deep learning

#optimizers — SGD, Momentum and Adam: the race to the minimum.
#batch-normalization — Batch normalization: keeping activations in the right range.
#rnn-lstm — RNN and LSTM: remembering a sequence.
#autoencoder — Autoencoder: compress then reconstruct.
#transfer-learning — Transfer learning: start from an already-trained network.
#gan — GAN: a forger against an inspector