Module 8 — Encoder-decoder for translation

Every module so far has kept the same red thread: a numeric sequence with a fixed lookback and a fixed horizon. Translation breaks both. The input is a French sentence of, say, 7 tokens; the output is an English sentence of 6 or 9 tokens; the alignment is not step-by-step. This module introduces the encoder-decoder architecture (Sutskever, Vinyals and Le, 2014), the first genuinely sequence-to-sequence design, and shows the exact ingredients — context vector, teacher forcing, start-of-sentence and end-of-sentence tokens, greedy decoding — that make it work.

The dataset for this module is a small corpus of short French-English sentence pairs from a public sentence collection (a subset of about 10 000 pairs, filtered to sentences under 12 tokens). Big enough to learn something, small enough to train in a few minutes on a laptop.

The idea in one paragraph

The encoder is an RNN (LSTM or GRU) that reads the source sentence one token at a time and finishes with a hidden state that summarises the whole sentence. This final state is the context vector: a single fixed-size vector meant to hold everything the decoder needs. The decoder is another RNN, initialised with that context, that generates the target sentence one token at a time, feeding its own previous output back as input.

Diagrammatically: [bonjour, le, monde] → encoder → context → decoder → [<sos>, hello, world, <eos>].

Special tokens: <sos>, <eos> and <pad>

Three special tokens are necessary. <sos> (start of sentence) is fed as the first input to the decoder so it has something to start from. <eos> (end of sentence) is the target the decoder learns to produce when the translation is complete; inference stops when it is generated. <pad> (padding) fills short sequences to a common length inside a batch; module 9 covers it in detail.

SOS, EOS, PAD, UNK = "<sos>", "<eos>", "<pad>", "<unk>"

def encode(sentence: str, vocab: dict[str, int], add_sos_eos: bool) -> list[int]:
    tokens = sentence.lower().split()
    ids = [vocab.get(t, vocab[UNK]) for t in tokens]
    if add_sos_eos:
        ids = [vocab[SOS]] + ids + [vocab[EOS]]
    return ids

The source sentence usually does not need <sos> or <eos>; the encoder simply reads it and stops. The target sentence needs both: <sos> at the input, <eos> at the output. This asymmetry is easy to miss and produces a decoder that either never starts or never stops.

Teacher forcing: the training trick that saves everything

The decoder is autoregressive: at step $t$, it uses its own prediction at $t-1$ as input. During inference this is unavoidable. During training it would be a disaster: an early mistake propagates through the entire sentence and the loss becomes noise.

Teacher forcing feeds the true previous target as input to the decoder at every step, rather than its own prediction. If the target sentence is <sos> hello world <eos>, the decoder is trained to:

• read <sos> → predict hello
• read hello (the true one, not the model's guess) → predict world
• read world (the true one) → predict <eos>

Training loss becomes a straightforward token-level cross-entropy. The mismatch with inference — where the decoder must feed its own predictions back — is called exposure bias and is a real, but manageable, limitation of teacher forcing.

# During training, both are shifted versions of the same target
decoder_input  = target[:, :-1]   # <sos> hello world
decoder_target = target[:, 1:]    # hello world <eos>

Building the model in Keras

The Functional API (course 08) makes the two-network structure explicit:

from tensorflow.keras import layers, Model

vocab_src, vocab_tgt = 8000, 8000
embed_dim, units     = 128, 256

# Encoder
enc_in  = layers.Input(shape=(None,), name="src")
enc_emb = layers.Embedding(vocab_src, embed_dim, mask_zero=True)(enc_in)
_, state_h, state_c = layers.LSTM(units, return_state=True)(enc_emb)
context = [state_h, state_c]

# Decoder
dec_in  = layers.Input(shape=(None,), name="tgt_shifted")
dec_emb = layers.Embedding(vocab_tgt, embed_dim, mask_zero=True)(dec_in)
dec_seq, _, _ = layers.LSTM(units, return_sequences=True,
                            return_state=True)(dec_emb, initial_state=context)
dec_out = layers.Dense(vocab_tgt, activation="softmax")(dec_seq)

model = Model([enc_in, dec_in], dec_out)
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")

The encoder returns only its final states, which become the initial state of the decoder. The decoder returns a full sequence of hidden states, one per target token, projected through a dense layer to the target vocabulary.

Greedy decoding at inference

Inference is fundamentally different from training because the decoder no longer receives the true target. The simplest strategy is greedy decoding: at each step, pick the token with the highest probability, feed it back as the next input, stop when <eos> appears.

def translate(model, src_ids, vocab_tgt, max_len=20):
    inv = {i: t for t, i in vocab_tgt.items()}
    src = np.array([src_ids])
    # Reuse the encoder half of the model
    enc = tf.keras.Model(model.get_layer("src").input,
                         model.get_layer("lstm").output)
    _, h, c = enc(src)

    out_ids = [vocab_tgt["<sos>"]]
    for _ in range(max_len):
        step_in = np.array([[out_ids[-1]]])
        dec_seq, h, c = decoder_step(step_in, [h, c])   # a small helper
        next_id = int(np.argmax(dec_seq[0, -1]))
        if next_id == vocab_tgt["<eos>"]:
            break
        out_ids.append(next_id)
    return " ".join(inv[i] for i in out_ids[1:])

Beam search is the next step up: instead of committing to the top-1 token at each position, keep the $k$ best partial hypotheses and expand each one. It buys 1 to 3 BLEU points at the cost of $k$ times more computation. For a course, greedy decoding is enough to see the model work.

The bottleneck that motivates attention

Read the architecture again with a critical eye: everything the decoder knows about the source sentence sits in the single fixed-size context vector. For a 3-word sentence that is generous; for a 30-word sentence it is a bottleneck. Empirically, translation quality degrades sharply as source length grows past 15 to 20 tokens.

Bahdanau et al. (2015) proposed the solution now called attention: at every decoding step, let the decoder look back at all the encoder's hidden states and weight them by relevance. The context becomes dynamic, not fixed. This is the birth of the attention mechanism, which grew into the Transformer and is the subject of course 12.

For this course, the encoder-decoder without attention is the historical landing point: it works well on short sentences, it makes the architecture crystal clear, and it explains — by contrast — what attention buys.

A translation model that always produces the same sentence has usually collapsed on <eos>

If the decoder emits <eos> immediately at every input, the target sequences were probably encoded without a leading <sos>, or the shifted arrays are misaligned. Print a training batch of (decoder_input, decoder_target) before starting a long run.

Overfit on 100 pairs before scaling

An encoder-decoder that cannot memorise 100 sentence pairs after 200 epochs has a bug — token IDs off by one, teacher forcing not applied, <sos> missing. Fix it there, not on 10 000 pairs where the signal is noisier.

In summary

• Encoder-decoder uses two RNNs: the encoder compresses the source into a context vector (its final states), the decoder generates the target autoregressively from that context.
• Special tokens <sos> and <eos> anchor the start and end of generation; teacher forcing feeds the true previous target during training to avoid compounding errors.
• Greedy decoding picks the top-1 token at each step; beam search improves quality at extra cost.
• The single fixed-size context is a bottleneck that limits translation quality on long sentences and motivates the attention mechanism introduced in course 12.

Next module: back to the electricity red thread, this time to face the practical questions of padding, masking and batch construction that any real recurrent pipeline has to solve.