Module 3 — Bag of words, TF-IDF and their limits

You have clean tokens (module 1) and a stable tokenization scheme (module 2). Turning them into numbers a classifier can read is the next step, and the oldest trick in the book — bag of words with a TF-IDF weighting — is still the honest baseline. Half of the projects that add a transformer would have shipped in one afternoon with the four lines below, and never noticed.

The bag: a sparse matrix

Take the 10 000 English reviews and their vocabulary of about 25 000 words. A bag-of-words matrix has 10 000 rows and 25 000 columns, with a cell counting how often each word appears in each review. On average, a review is 40 words long: at most 40 non-zero cells out of 25 000. The matrix is 99.8 % zeros.

If you store it as a dense NumPy array, it costs 2 GB and 90 % of the memory holds zeros. A sparse matrix (scipy.sparse.csr_matrix) stores only the non-zero entries and the arithmetic on them stays linear in the number of non-zeros. Everything scikit-learn does with text uses sparse matrices under the hood; you almost never have to think about it, provided you never call .toarray() in a moment of curiosity.

from sklearn.feature_extraction.text import CountVectorizer

vect = CountVectorizer(min_df=2, max_df=0.9)
X_train = vect.fit_transform(train_texts)   # sparse (n_docs, vocab)
print(X_train.shape, X_train.nnz)           # (10000, ~15000), ~400000

Two parameters do most of the vocabulary trimming. min_df=2 drops words appearing in fewer than two documents — that removes typos, one-off product names, most of the long tail. max_df=0.9 drops words appearing in more than 90 % of documents; on this corpus that means the, and, a, is, all the words a stop list would have removed. These two lines replace an explicit stop-word list and are less brittle across languages.

n-grams put a hint of context back

The bag throws away word order: not good and good not are the same vector. A cheap fix is to add 2-grams as columns. Set ngram_range=(1, 2) and the vocabulary now contains not, good, not good, good not. The vocabulary triples; the accuracy on sentiment usually gains 1 to 3 points because the negations that module 1 saved from the stop list now carry through.

Going to 3-grams doubles again the vocabulary for a much smaller gain. In practice, (1, 2) is the sweet spot on short texts like reviews, and (1, 3) on longer documents.

From raw counts to TF-IDF

Raw counts have a problem: a word that appears in almost every review carries little discriminative power, even if it is frequent. TF-IDF fixes this with two multiplied terms.

The term frequency is how often the word appears in the document, sometimes scaled by document length. The inverse document frequency is a logarithm of the inverse fraction of documents containing the word. Formally, for a term $t$ appearing in $\mathrm{df}(t)$ documents out of $N$:

$\mathrm{idf}(t) = \log \frac{N + 1}{\mathrm{df}(t) + 1} + 1$

$\mathrm{tfidf}(t, d) = \mathrm{tf}(t, d) \times \mathrm{idf}(t)$

A word that appears in every document has $\mathrm{df}(t) = N$, so $\mathrm{idf}$ is close to 1: no boost. A rare word that appears in ten documents out of ten thousand has an IDF around 7: its counts are multiplied by 7. Bag-of-words gives every word the same weight per occurrence; TF-IDF says a rare hit is worth more than a common one.

The baseline in four lines

Combine TfidfVectorizer with a logistic regression. You now have a classifier you must beat before you talk about anything more expensive.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

baseline = Pipeline([
    ("tfidf", TfidfVectorizer(min_df=2, max_df=0.9, ngram_range=(1, 2))),
    ("clf", LogisticRegression(max_iter=1000, C=1.0)),
])

baseline.fit(train_texts, train_labels)
print(baseline.score(test_texts, test_labels))

On the ten-thousand-review sentiment task (5-star ratings collapsed to positive versus negative), expect this baseline to land between 0.87 and 0.90 in accuracy. It fits in seconds on a laptop and predicts in microseconds per document. It is what you have to beat.

Log-odds inspection

LogisticRegression learns one coefficient per word. Sorting them gives you the words most predictive of each class — terrible, waste, disappointed on one side, love, perfect, recommend on the other. This inspection is free and gives you a first sanity check that will save you from a hundred embarrassing mistakes later.

What the baseline cannot see

The bag is deliberately memoryless. Three failure modes are worth naming.

Word order. The battery was terrible but the screen is amazing and The screen is terrible but the battery was amazing are the same vector once you strip stop words. TF-IDF cannot separate them; only a model that reads the sentence sequentially can.

Meaning. phone, handset, smartphone, mobile are four different columns with no relationship in the matrix. A review that uses handset throughout gains no benefit from the phone reviews used in training. Word embeddings (module 4) fix this by giving similar words nearby vectors.

Ambiguity. apple in the apple was crisp and apple released a new phone is the same column. TF-IDF sees the frequency, not the meaning. Contextual embeddings (module 5) are what start to separate the two.

Do not fit the vectorizer on train + test

TfidfVectorizer learns the vocabulary and the IDFs from the training corpus. If you fit it on the concatenation of train and test, the IDFs peek at the test set and you overestimate accuracy by 1 to 2 points. Always call fit on training texts only, then transform on the test.

When the baseline is enough to ship

For topic classification, spam versus ham, language identification and any task where the label correlates with a small set of discriminative words, TF-IDF plus a linear model matches or beats a fine-tuned transformer more often than you would think. It also runs on a CPU, without a GPU quota, at 100 000 predictions per second. Reserve the heavy artillery for tasks where meaning and context are the whole game: fine-grained sentiment with irony, paraphrase detection, question answering, entailment.

In summary

• The bag of words is a sparse matrix with one column per vocabulary token; a review touches at most a few dozen of the 25 000 columns.
• TF-IDF downweights very common words and boosts rare, discriminative ones; combined with a linear model it gives a strong baseline in four lines.
• n-grams put a limited notion of order back; (1, 2) is the practical sweet spot on short reviews.
• The baseline cannot see word order, synonymy or word sense; those are the three doors the rest of the course walks through.

Next module: dense word embeddings — the moment phone and handset finally acquire nearby vectors.