Module 6 — The encoder: BERT and understanding models
Module 5 finished the encoder layer. Stack of them and you have the Transformer encoder. Add a masked pre-training objective on top of a very large corpus and you have BERT, the model that redefined the NLP leaderboard in 2018. This module builds the encoder from our red-thread parts, then explains how BERT is trained and adapted to real tasks.
The encoder is just a stack
import torch.nn as nn
from embedding import TokenAndPositional
from encoder_layer import EncoderLayer
class Encoder(nn.Module):
def __init__(self, vocab_size, d_model=512, num_heads=8, num_layers=6):
super().__init__()
self.embed = TokenAndPositional(vocab_size, d_model)
self.layers = nn.ModuleList(
[EncoderLayer(d_model, num_heads) for _ in range(num_layers)]
)
self.norm = nn.LayerNorm(d_model)
def forward(self, ids, mask=None):
x = self.embed(ids)
for layer in self.layers:
x = layer(x, mask)
return self.norm(x)
Six encoder layers is what BERT-base uses on the encoder side; twelve is BERT-large. Save the file as encoder.py; module 10 wires it into the full model.
Two properties define an encoder as opposed to a decoder.
- Bidirectional attention. Any position can attend to any other, in both directions. That is the "encoder" way of reading a sentence: everything, at once.
- One output per input token. The encoder emits a same-length sequence of contextualised representations. What you do with them depends on the downstream task.
Masked language modelling: the training objective
BERT is trained from scratch on Wikipedia and BookCorpus with no labels. The trick is to fabricate a task from the text itself.
- Mask about 15 % of tokens in each sentence by replacing them with a special
[MASK]token. - The model receives the corrupted sentence and must predict the original tokens at the masked positions.
- The loss is a cross-entropy over the vocabulary at each masked position; the rest of the positions do not contribute.
That is masked language modelling (MLM). It forces the encoder to combine left and right context, because a masked token in the middle is only predictable from both sides. Contrast this with the causal objective GPT uses in the next module: the encoder objective requires bidirectionality.
BERT adds a second, weaker task called next sentence prediction: two sentences are concatenated, and the model predicts whether the second follows the first. RoBERTa dropped this task and matched or exceeded BERT's results, which showed it was not essential.
BERT does not mask exactly 15 % with [MASK]. It replaces 80 % of the chosen tokens by [MASK], 10 % by a random token, and leaves 10 % unchanged. The model therefore cannot rely on "there is a [MASK] here" as its only signal, and must learn robust representations. Getting the ratios wrong (a common mistake in custom pre-training) hurts final quality by several points.
The [CLS] and [SEP] special tokens
BERT prepends a [CLS] (classification) token to every input and inserts a [SEP] between sentence pairs. These tokens participate in attention like any other. Their special role is what happens at the output:
- The final hidden state at position
[CLS]is taken as a sentence-level representation. Any classification head reads from that one vector. - The final hidden states at content positions are read for token-level tasks — named entity recognition, extractive question answering, part-of-speech tagging.
The [CLS] vector is not privileged during pre-training; classification uses it because it is a convenient fixed slot that has attended to every other position. Some tasks work better with an average of all token vectors, or with a pooler layer on top of [CLS]; the default is [CLS] because it is simple and reproducible.
Fine-tuning: a task head on top
Adapting BERT to your task is remarkably light-touch. You add a small head — one or two layers — on top of the frozen or partly-frozen encoder, and train on a labelled dataset.
from transformers import AutoModel, AutoTokenizer
import torch.nn as nn
class BertClassifier(nn.Module):
def __init__(self, backbone="bert-base-uncased", num_classes=2):
super().__init__()
self.encoder = AutoModel.from_pretrained(backbone)
self.head = nn.Linear(self.encoder.config.hidden_size, num_classes)
def forward(self, input_ids, attention_mask):
out = self.encoder(input_ids=input_ids, attention_mask=attention_mask)
cls = out.last_hidden_state[:, 0]
return self.head(cls)
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
batch = tok(["Great movie!", "Do not watch."], padding=True, return_tensors="pt")
model = BertClassifier()
logits = model(**batch)
A few good practices worth writing down:
- Train on the whole model, not just the head, at a very low learning rate (2e-5 to 5e-5). Freezing the encoder saves compute but loses one to two accuracy points.
- Use linear warmup and decay: the warmup that pre-norm did not require here becomes essential again because the pre-trained weights are already well-scaled and would be disturbed by large early updates.
- Fine-tune for 2 to 4 epochs, not more. BERT overfits small datasets very quickly.
For extraction (span selection, named-entity recognition), the head reads the sequence of token outputs and predicts a label per token — start / end for question answering, one class per token for NER. [CLS] plays no special role there.
The BERT family
Once BERT existed, everyone published a variant. The important ones on the encoder side are:
| Model | What it changes | When to prefer |
|---|---|---|
| RoBERTa | more data, longer training, no NSP, dynamic masking | almost always a better starting point than BERT |
| DistilBERT | halved depth via distillation, 40 % smaller, 60 % faster | inference-time constraints |
| ALBERT | shared parameters across layers, factorised embedding | large models on small compute |
| DeBERTa | disentangled attention, relative positions | leaderboard-chasing tasks |
| CamemBERT / FlauBERT | French text | French text |
| XLM-R | 100 languages in a single model | multilingual, low-resource |
For your project, the practical rule is: start with roberta-base for English, or a language-specific model for anything else, and only escalate if metrics demand it. bert-base-uncased remains a good baseline for education because every tutorial ever written uses it.
Where the encoder wins and where it does not
An encoder produces a rich representation but does not generate. It cannot, by construction, be handed a prompt and asked to continue. The two families of tasks that suit an encoder are:
- Classification: sentiment, spam, intent, toxicity, topic. One label per input, or one per token.
- Extraction: named entities, spans, relations, retrieval-style scoring where you compare two encoder outputs.
For generation — translation, summarisation, chat — you need at least a decoder, which is the subject of module 7. Trying to shoehorn generation into an encoder ends up with awkward mask-filling loops that are always worse than a proper decoder on the same budget.
Almost no one trains BERT from scratch anymore. Downloading a pre-trained checkpoint and fine-tuning it on your labels costs a few hundred euros in compute versus a few million to redo the pre-training. Choose the smallest model that matches your accuracy target — DistilBERT is often enough — and only move up when the ceiling is measured.
In summary
- The Transformer encoder stacks of the layers built in module 5 and produces one contextualised vector per input token, with bidirectional attention.
- BERT is trained by masked language modelling on a large unlabelled corpus, using
[CLS]for classification and content tokens for extraction. - Fine-tuning is light: a small head on top of the encoder, trained end-to-end at a very low learning rate for a handful of epochs.
- Variants — RoBERTa, DistilBERT, DeBERTa, CamemBERT — trade data, depth or attention design; RoBERTa is a safer default than BERT itself.
Next module: the decoder, which flips the causal mask and gives us GPT — a family built for generation instead of understanding.