Module 7 — Named entity recognition

Classifying a whole review as positive or negative (module 6) is one thing; pulling out what the customer talks about is another. A product team does not need to know that 20 % of reviews are negative — it needs to know that half of those mention battery, charger and MagSafe by name. Named entity recognition (NER) is the tool for that.

What we want out of the reviews

Given a review, produce a list of spans, each tagged with a type:

Text  : "I bought the AirPods Pro last month in Berlin, and the case cracked already."
Spans : [
  (13, 24, PRODUCT),   # AirPods Pro
  (40, 46, LOC),       # Berlin
  (56, 60, PART),      # case  — optional, task-specific
]

A production system exposes those spans as coloured highlights in a review dashboard, and as filters in an internal search. It also feeds them back into module 6: battery mentions correlate with 1-star ratings differently from screen mentions.

BIO tagging: the standard encoding

NER is not a classification of the whole sentence; it is a classification of each token. The encoding that makes this tractable is BIO (Begin, Inside, Outside):

Token	BIO tag
I	O
bought	O
the	O
AirPods	B-PROD
Pro	I-PROD
last	O
month	O
in	O
Berlin	B-LOC

B-PROD marks the start of a product entity, I-PROD marks a continuation, O marks a non-entity. Every token receives exactly one tag. The variant IOB2 is what modern libraries use; there is also BILOU which adds L (last) and U (unit), slightly more expressive, marginally harder to train.

The alignment problem no tutorial explains at first

Here comes the pitfall specific to subword tokenisers (module 2). Your training labels are on words: AirPods → B-PROD. But the tokenizer breaks AirPods into Air and ##Pods. The model sees two tokens; you have one label.

Get this wrong and you either train the model on shifted labels — every subword after the first receives the label of an unrelated word — or you throw an exception halfway through the first batch.

The canonical fix in Transformers is word_ids, which reports which original word each subword came from.

def tokenize_and_align(examples):
    encoded = tok(examples["tokens"], is_split_into_words=True,
                  truncation=True, max_length=256)
    aligned = []
    for i, labels in enumerate(examples["ner_tags"]):
        word_ids = encoded.word_ids(batch_index=i)
        previous_word = None
        new_labels = []
        for wid in word_ids:
            if wid is None:                       # special tokens [CLS] [SEP]
                new_labels.append(-100)
            elif wid != previous_word:            # first subword of a word
                new_labels.append(labels[wid])
            else:                                 # subsequent subword
                new_labels.append(-100)           # ignored in the loss
            previous_word = wid
        aligned.append(new_labels)
    encoded["labels"] = aligned
    return encoded

Two conventions matter here. -100 is the value PyTorch's CrossEntropyLoss ignores; it lets you keep every subword in the input while only backpropagating on the first subword of each word. The alternative — propagating the label to every subword — trains the model to predict I-PROD on ##Pods, which is fine on paper and often hurts by a small margin in practice.

Fine-tuning an encoder for NER

The architecture is a pretrained encoder with a token classification head: one linear layer applied to every position's contextual vector, outputting a distribution over the tag set.

from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer, DataCollatorForTokenClassification

model = AutoModelForTokenClassification.from_pretrained(
    "distilbert-base-cased",
    num_labels=len(tag_list),
    id2label={i: t for i, t in enumerate(tag_list)},
    label2id={t: i for i, t in enumerate(tag_list)},
)

args = TrainingArguments(
    output_dir="ner-out", num_train_epochs=3, learning_rate=3e-5,
    per_device_train_batch_size=16, evaluation_strategy="epoch",
    logging_steps=50, weight_decay=0.01,
)

trainer = Trainer(
    model=model, args=args,
    train_dataset=train_ds, eval_dataset=dev_ds,
    data_collator=DataCollatorForTokenClassification(tok),
    tokenizer=tok, compute_metrics=compute_ner_metrics,
)
trainer.train()

Two implementation notes. The cased variant (distilbert-base-cased) is preferred: Berlin versus berlin and Apple versus apple are cues NER cannot afford to lose. And DataCollatorForTokenClassification handles the padding of both inputs and labels to the same length in each batch — a common source of runtime crashes if you build it yourself.

The right metrics: per entity, not per token

If you evaluate NER token by token, an O prediction on an easy word inflates accuracy dramatically: reviews are 90 % non-entities, so a model predicting O everywhere gets 90 % token accuracy. Useless.

The standard is entity-level F1: an entity counts as correct only if its span and its type both match the reference. Compute this with seqeval.

from seqeval.metrics import classification_report as seq_report
from seqeval.metrics import f1_score

def compute_ner_metrics(eval_pred):
    logits, labels = eval_pred
    preds = logits.argmax(-1)
    y_true, y_pred = [], []
    for pred_seq, label_seq in zip(preds, labels):
        true = [tag_list[l] for l in label_seq if l != -100]
        pred = [tag_list[p] for p, l in zip(pred_seq, label_seq) if l != -100]
        y_true.append(true); y_pred.append(pred)
    return {"f1": f1_score(y_true, y_pred)}

seq_report(y_true, y_pred) breaks the score by entity type. This is where insight lives: it is common to see 0.90 F1 on LOC and 0.55 F1 on PROD on a product review corpus, because place names are shared with a general newswire corpus while product names are corpus-specific.

Common failure modes

Consistency violations. The model outputs O, I-PROD, I-PROD — a continuation without a beginning. A CRF layer used to fix this at the cost of complexity; a post-processing pass that turns any I-X after O or B-Y (Y ≠ X) into B-X is simpler and about as effective. Modern encoders make these violations rare, and seqeval implicitly forgives most of them.

Boundary errors. The model predicts MagSafe charger as one entity when the annotation says MagSafe alone. Entity-level F1 penalises this as harshly as missing the entity entirely. Reviewing the boundary decisions of the annotation guide before training saves hours later.

Rare types. An entity type that appears 20 times in training will not be learned well by any encoder. Merging it into a broader type (ORG covering brands and companies) is often better than pretending precision at 5 examples per split.

Do not merge B- and I- into a single "in-entity" tag

The temptation is real: it halves the number of tags and looks simpler. It also loses the ability to distinguish two adjacent entities of the same type. iPhone iPhone charger, in a review, describes two separate items; without the B- / I- distinction the tagger cannot represent that.

Off-the-shelf NER before training your own

Before spending time on annotation, run spaCy's en_core_web_trf or a Hugging Face dslim/bert-base-NER on 500 reviews and look at the output. Half of the entities you care about (LOC, ORG, PERSON) are often already extracted well; you only need to train for the domain-specific ones (PROD, PART).

In summary

• BIO tagging turns entity extraction into a token classification problem, with one label per token and a B- / I- distinction that separates adjacent entities.
• The token–label alignment with subword tokenizers is where most implementations break; use word_ids and mask everything but the first subword of each word.
• Evaluate at the entity level, not the token level: seqeval reports per-type precision, recall and F1 — token accuracy hides everything you care about.
• Domain-specific entities (product names, parts) usually require in-domain training, while generic entities (LOC, ORG, PERSON) work well off the shelf.

Next module: summarising the reviews and answering questions on them, and why ROUGE will convince you a bad summariser is good.