Module 4 — Encoding: one-hot, ordinal, target
Models consume nothing but numbers. Yet real data is full of categories: city, brand, postal code, contract type. Turning them into numbers seems innocuous — it is in fact one of the weightiest decisions in feature engineering, and the wrong choice silently injects false information.
The trap of naive encoding
The immediate temptation: number the categories. Paris = 1, Lyon = 2, Marseille = 3. The problem appears as soon as you think about it: the model reads ordered numbers. It infers that Marseille > Lyon > Paris, that the gap between Paris and Marseille is twice that between Paris and Lyon, and that an average of Paris and Marseille yields Lyon.
None of these statements makes sense, yet they are mathematically written into the data. A linear model, which multiplies the variable by a coefficient, will rely on them. Arbitrarily numbering unordered categories is a mistake, not a shortcut.
One-hot: the reference for low cardinality
One-hot encoding creates one binary column per category: the column of the present category is 1, the others 0. No order is suggested any more, and each category gets its own coefficient.
from sklearn.preprocessing import OneHotEncoder
OneHotEncoder(handle_unknown="ignore", drop="first", sparse_output=False)
Two parameters genuinely matter. handle_unknown="ignore" avoids a crash in production when a category absent from training shows up — a mundane and otherwise fatal situation. drop="first" removes a column that has become redundant (if it is none of the others, it is necessarily the first); useful for linear models, where perfect collinearity is a nuisance, and useless for trees.
Its limitation is structural: cardinality. A postal-code variable with 6,000 values produces 6,000 columns. The matrix explodes, distances lose their meaning (the curse of dimensionality), each column contains almost nothing but zeros, and rare categories are seen too seldom to be learned. In practice, one-hot works up to a few dozen categories.
Ordinal encoding: when order genuinely exists
Some categories are genuinely ordered: "low < medium < high", "primary < secondary < higher". Numbering is then not only legitimate but desirable, since the order is true information you pass to the model.
from sklearn.preprocessing import OrdinalEncoder
OrdinalEncoder(categories=[["low", "medium", "high"]])
One point of method: specify the order explicitly. Left free, the encoder sorts alphabetically and produces a false order — "high, low, medium" — reintroducing exactly the problem you thought you had avoided.
High cardinality: target and frequency encoding
For variables with many levels, two approaches usefully replace one-hot.
Frequency encoding replaces the category by its number of occurrences. A single column, no leakage, and often relevant information — a product code seen 10,000 times does not have the same status as one seen three times.
Target encoding replaces the category by the mean of the target for that category. A city where 8% of customers churn becomes 0.08. This is very powerful: a single column, information directly tied to what you predict, and the ability to handle thousands of levels.
That power has a downside, and it must be named clearly: target encoding uses the target, therefore it leaks by construction. A category appearing only three times is assigned the mean of those three observations — the model memorizes those rows instead of learning. Two indispensable remedies:
- smoothing: blend the category mean with the global mean, weighted by count, so that rare categories tend toward the overall mean;
- out-of-fold computation: compute the mean by cross-validation, each fold being encoded with statistics from the other folds.
The category_encoders library provides TargetEncoder with built-in smoothing, and scikit-learn now offers its own TargetEncoder with internal cross-validation. Hand-rolling this is an excellent way to manufacture a leak.
Up to about ten categories, one-hot without hesitation. From ten to fifty, one-hot remains viable, possibly after grouping rare levels into an "Other". Beyond that, frequency or target with smoothing. And if the model is a gradient boosting, note that LightGBM and CatBoost handle categorical variables natively — often better than manual encoding, particularly at high cardinality.
Summary
- Numbering unordered categories injects a false order that linear models will exploit.
- One-hot is the reference at low cardinality;
handle_unknown="ignore"protects production, but the method explodes beyond a few dozen levels. - Ordinal encoding only suits genuine orders, and the order must be declared explicitly.
- At high cardinality, use frequency or target encoding; the latter leaks by construction and requires smoothing and out-of-fold computation.
Next module: dates and cyclical features, where apparently simple information calls for a particular encoding.