Module 6 — Text features: bag of words and n-grams

Customer reviews, product descriptions, support tickets: text is everywhere, and no tabular model consumes it directly. This module gives the classical representations, which remain excellent baselines — often hard to beat, always fast and interpretable, unlike the embedding approaches the language courses will cover.

Normalize first

Before any vectorization, reduce the variations that contribute nothing. "Excellent", "EXCELLENT" and "excellent!" mean the same thing and should become so in the data.

The usual operations: lowercase, strip punctuation, normalize whitespace. Two choices, however, require a considered decision.

Stop words ("the", "of", "and") clutter without discriminating, and are usually removed. But not always: in sentiment analysis, the negation "not" is decisive, and removing it inverts the meaning of the sentence. The stop-word list is chosen according to the task, never by blind default.

Stemming and lemmatization reduce inflected forms to a common base, so that "running", "ran" and "runs" count together. Stemming truncates crudely, lemmatization uses a dictionary and produces a linguistically correct result; it is slower but more reliable.

Bag of words

The idea of the bag of words is to count occurrences: each word of the vocabulary becomes a column, each document a row of counts. You lose word order — hence the name — but you obtain a usable numeric matrix.

from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer(
    lowercase=True,
    max_features=5000,     # keep only the 5000 most frequent words
    min_df=5,              # ignore words seen in fewer than 5 documents
    max_df=0.8,            # ignore words present in more than 80% of documents
)
X = vec.fit_transform(texts)

The three filtering parameters do most of the work. min_df eliminates typos and terms too rare to be learned; max_df eliminates ubiquitous, hence non-discriminating, words; max_features bounds the dimension. Without them, the matrix reaches tens of thousands of near-empty columns.

TF-IDF: weighting by rarity

Raw counting has a flaw: a word frequent in every document weighs heavily without distinguishing anything. TF-IDF weighting corrects this by combining two quantities: the frequency of the term in the document (TF) and the inverse of its document frequency (IDF), that is, its rarity in the corpus.

$$\text{tfidf}(t, d) = \text{tf}(t, d) \times \log\frac{N}{\text{df}(t)}$$

A word that appears often in this document but rarely elsewhere gets a high weight: that is precisely a characteristic word. Conversely, a word present everywhere sees its weight collapse.

from sklearn.feature_extraction.text import TfidfVectorizer
X = TfidfVectorizer(max_features=5000, min_df=5, ngram_range=(1, 2)).fit_transform(texts)

TF-IDF is the default choice for text classification, and it almost always outperforms raw counting.

N-grams: recovering some word order

Bag of words ignores order, which poses a real problem: "not good" and "good" share the word "good". N-grams answer by counting sequences of consecutive words. With ngram_range=(1, 2), you count single words and pairs: "not good" becomes a feature in its own right, distinct from "good".

The cost is an explosion of the vocabulary, hence the importance of min_df and max_features. In practice, bigrams bring a clear gain, trigrams rarely enough to justify the extra dimension.

Descriptive features, too often forgotten

Before vectorizing, a few simple features compute in one line and frequently prove highly predictive:

df["n_chars"]        = df["text"].str.len()
df["n_words"]        = df["text"].str.split().str.len()
df["n_uppercase"]    = df["text"].str.count(r"[A-Z]")
df["n_exclamations"] = df["text"].str.count("!")

The length of a review, the proportion of uppercase or the number of exclamation marks carry a real signal — about dissatisfaction, about the spam nature of a message. These features cost almost nothing and slot in alongside existing tabular features, where a 5,000-column TF-IDF matrix integrates far less easily.

Where this module stops

Bag of words and TF-IDF treat words as independent symbols: "car" and "automobile" remain two unrelated columns. Embedding and transformer representations, the subject of courses 12 and 13, capture that semantic proximity. But start here: on modest corpora and for a simple classification task, TF-IDF plus logistic regression is a fast, interpretable baseline, and frequently very close to heavyweight approaches.

Summary

• Normalize first (lowercase, punctuation); stop words and lemmatization are decided by task, negation possibly being essential.
• Bag of words counts occurrences and loses order; min_df, max_df and max_features control the dimension.
• TF-IDF weights by rarity in the corpus and highlights characteristic words: it is the default choice.
• N-grams recover part of the word order ("not good"); and simple descriptive features are cheap and often predictive.

Next module: features from aggregations and time windows, where information is built from several rows at once.