Module 1 — Text processing pipeline: cleaning and normalization

Course 04 taught you to distrust dirty tabular data. In text the picture is worse: two documents can be visually identical and byte-for-byte different, and one over-eager cleaning line can silently delete the very words that carry the label. This module fixes the ground rules on the corpus of ten thousand English customer reviews you will use throughout the course.

Two documents that look the same, and are not

Load a sample of the reviews and print the first character of a word that visibly starts with e.

import unicodedata

a = "café"                       # "café" typed with a single character é
b = "cafe\u0301"                 # "cafe" plus a combining acute accent
print(a == b)                    # False
print(len(a), len(b))            # 4 5
print(unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b))  # True

The two strings render as café on any screen, yet they are not equal. This is called a canonical equivalence problem, and it appears constantly on user-generated text: web forms, mobile keyboards and OCR pipelines produce a mix of precomposed (é) and decomposed (e + combining ́) forms. Any deduplication, groupby, join, or string comparison downstream will treat them as different.

The fix is to run Unicode normalization at the very entry of the pipeline. Two common forms cover the vast majority of cases:

• NFC recomposes decomposed sequences and preserves the visual and semantic identity of the text.
• NFKC goes further: it maps compatibility characters to their canonical equivalent, so the ligature fi becomes fi and the full-width Ａ becomes A.

NFC is the safe default. NFKC is stronger but destructive on markup-sensitive content — it changes superscripts, subscripts and certain currency signs — so reserve it for downstream text indexing where you want to consider variants as one form.

The English corpus, in one pass

Here is a small function you will keep enlarging as the course progresses. It is deliberately explicit: every step is one line so you can turn it off individually when it hurts more than it helps.

import re
import unicodedata

WHITESPACE = re.compile(r"\s+")
URL = re.compile(r"https?://\S+|www\.\S+")

def clean_review(text: str) -> str:
    text = unicodedata.normalize("NFC", text)
    text = URL.sub(" ", text)                  # keep the space
    text = text.replace("\u00a0", " ")         # non-breaking space
    text = WHITESPACE.sub(" ", text).strip()
    return text

Run it on a hundred reviews and diff the output against the input. You will see three families of noise disappear: stray whitespace, invisible non-breaking spaces (they wreck token boundaries), and URLs, which almost never help classification and often become their own class in a bag-of-words model.

What must not be stripped

The default reflex is to lowercase and remove punctuation. Both are useful and both destroy signal on this corpus.

Casing. Lowercasing Apple and apple merges the brand with the fruit. On product reviews it also merges LG with lg, iOS with ios, and any acronym with its own initials. If you are going to use a bag-of-words baseline (module 3), keeping case reveals brand mentions that turn out to be predictive of the star rating. If you are going to use a case-insensitive pretrained model (BERT-base-uncased, module 5), lowercasing is done for you, in a way you cannot undo — so the cased and uncased branches of the course diverge here.

Punctuation. A brutal re.sub(r"[^\w\s]", "", text) deletes contractions (don't becomes dont), it turns state-of-the-art into stateoftheart, and — the one that hurts the most on customer reviews — it deletes every exclamation and question mark. Yet ! and ? correlate strongly with 1-star and 5-star reviews in every corpus we have looked at. Keep punctuation as its own tokens.

Stripping accents is destructive normalization

On English the temptation is small; on user-generated content it is real, because reviews often contain foreign brand names (Häagen-Dazs, Nescafé) and words borrowed from other languages. Stripping accents merges pairs of distinct words in those languages and can flip the meaning of a sentence quoted in a review. If you are going to strip, do it in a searchable index copy, not in the training text.

Stop words: not free either

The classic move is to drop a fixed list of very common words: the, a, is, and, of, to. For a topic-detection task this is fine. For sentiment it is dangerous. Consider not good enough and good enough. If not sits on the stop list — and it does in the default NLTK English list — the two reviews collapse to the same three tokens and become indistinguishable.

The safe rule: for sentiment, either keep negations and intensifiers, or don't remove stop words at all. TF-IDF (module 3) already downweights very frequent tokens, so an explicit stop list adds little.

Lemmatization versus stemming

Both aim to fold morphological variants into a canonical form (running, ran, runs → run). They differ in method and in cost.

Approach	Method	Typical output on studies	Cost
Stemmer (Porter, Snowball)	pattern rewrites	studi	very cheap
Lemmatiser (spaCy, WordNet)	dictionary + POS tag	study	ten to a hundred times slower

The stemmer produces non-words; the lemmatiser preserves lookups. On a TF-IDF baseline on 10 000 reviews, expect a stemmer to shrink the vocabulary by roughly 20 % without moving accuracy either way. On a modern transformer, do neither: the subword tokenisers of module 2 already split studies into pieces the model has learned to recompose.

In summary

• Normalize Unicode first: NFC at entry aligns visually identical strings that differ in bytes; without it, deduplication and joins under-count reviews.
• Keep case, keep punctuation, be careful with stop words: on sentiment they carry signal, and their default removal turns opposite reviews into the same vector.
• Lemmatisation is a compromise between vocabulary shrinkage and cost; on a bag-of-words baseline it helps a little, on a pretrained encoder it hurts by breaking subword alignment.
• Every transformation is a decision you can defend: write cleaning as a chain of one-liners you can switch on and off, and diff a sample of the corpus before and after.

Next module: tokenization proper — how the same sentence becomes 12 pieces or 45 depending on the tokenizer, and why the difference matters for cost and quality.