Module 2 — Missing values: dropping, imputing, indicating
No real dataset is complete. A sensor fails, a customer refuses to disclose income, a form changes version. Since most algorithms reject missing values, you must decide what to do with them — and that apparently technical choice can introduce a bias no validation will reveal.
The first question is not "how to fill" but "why is it missing"
This is the reflex that separates correct treatment from tinkering. The mechanism of missingness determines what you are allowed to do:
| Mechanism | Meaning | Consequence |
|---|---|---|
| MCAR | missing completely at random | imputation is unbiased |
| MAR | missingness depends on other observed variables | imputation possible by relying on them |
| MNAR | missingness depends on the missing value itself | imputation biases; the missingness is informative |
The MNAR case is the most frequent and the most dangerous. If high earners are more likely to withhold their income, replacing missing values by the mean mechanically pulls the distribution downward, and the model learns a reality that does not exist. Here, the very fact that a value is missing carries information — hence the missingness indicator below.
In practice you do not prove the mechanism, you investigate it: compare complete and incomplete rows on the other variables, and ask the business how the data is collected. An explanation of the collection process beats a statistical test.
Strategy 1 — Drop
You can drop incomplete rows. Simple, assumption-free, but costly: with 5% missing spread over ten variables, you may lose a third of your observations. And above all, if missingness is not MCAR, you preferentially remove a certain profile and bias the sample.
You can also drop the column, which becomes reasonable beyond 50 or 60% missing — but not before checking that the missingness itself is not predictive.
Strategy 2 — Impute
Imputing means replacing with a plausible value. From simplest to most elaborate:
from sklearn.impute import SimpleImputer, KNNImputer
SimpleImputer(strategy="median") # numeric, robust to outliers
SimpleImputer(strategy="most_frequent") # categorical
SimpleImputer(strategy="constant", fill_value="Unknown") # explicit category
KNNImputer(n_neighbors=5) # from similar observations
The median is preferable to the mean as soon as extreme values are present, for the same reason as in the previous course. For a categorical variable, creating an "Unknown" category is often superior to mode imputation: you invent no information, and the model can learn a behavior specific to that group.
Model-based methods (KNNImputer, or iterative imputation) exploit correlations between variables and are more faithful when missingness is MAR. Their price: more computation, a risk of overfitting, and a transformation that is harder to explain.
All these methods share a structural limitation: they reduce the variance of the variable, since they concentrate values on a single estimate. Correlations then appear slightly stronger than they really are.
Strategy 3 — Add a missingness indicator
This is the answer to the MNAR case, and it is too rarely used. You impute the variable and create a binary column flagging that the value was absent:
SimpleImputer(strategy="median", add_indicator=True)
The model then has both pieces of information: a usable value, and the fact that it was reconstructed. If missingness is predictive — the customer who does not disclose income, the patient who did not take the test — the model can use it. The cost is one column; the gain, on real data, is frequently clear.
Computing the median on the whole dataset then splitting into train and test injects test information into training. The resulting score is optimistic and not reproducible. The median must be computed on training data alone, then applied to the test set. The structural remedy is the pipeline of module 10, which makes the mistake impossible.
Summary
- The first question is why the value is missing: MCAR, MAR or MNAR — the last being the most frequent and the most treacherous.
- Dropping is assumption-free but costly, and biases the sample as soon as missingness is not random.
- Impute by median, mode or an "Unknown" category; model-based methods suit the MAR case at the cost of simplicity, and any imputation reduces variance.
- The missingness indicator preserves the information carried by absence itself; and imputation is computed on training data alone, on pain of leakage.
Next module: scaling, the second basic transformation, and the cases where it is indispensable or useless.