Skip to main content

Module 9 — Feature selection: filters, wrappers, importance

Previous modules mostly added features. Through decomposing dates, encoding categories and aggregating histories, you end up with hundreds of columns, many contributing nothing. This module sorts them out — and the question is not merely aesthetic.

Why reduce​

Four reasons, two of which are often underestimated:

  • performance: useless features add noise the model can overfit, particularly when observations are few relative to the number of columns;
  • speed: faster training and prediction, which matters under real-time constraints;
  • interpretability: a 15-feature model can be explained to a decision-maker, a 400-feature model cannot;
  • operating cost, the most neglected: every feature is a computation to maintain in production, a data source that can fail, a dependency that can drift. A feature worth 0.1 performance point for an extra nightly pipeline is a bad deal.

The prerequisite: eliminate the obviously useless​

Before any elaborate method, two free cleanups. Near-constant features (a single value in 99% of cases) cannot discriminate anything. Near-duplicate features (correlation above 0.95 with another) count the same information twice; keep one, favoring the one that is simpler to produce and explain.

from sklearn.feature_selection import VarianceThreshold
VarianceThreshold(threshold=0.01)      # removes near-constant features

Filter methods: fast and model-agnostic​

A filter evaluates each feature by a statistical measure of association with the target, without training a model. This is very fast, which allows you to trim a set of several thousand columns.

from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif

SelectKBest(f_classif, k=50)             # statistical test: linear association
SelectKBest(mutual_info_classif, k=50)   # mutual information: any association

Mutual information is preferable to the F-test in most cases, because it also captures non-linear relationships, which the F-test ignores.

Filters do, however, have a weakness of principle worth knowing: they judge each feature in isolation. Two individually useless features can be decisive together, and a filter will discard both. Symmetrically, it will keep ten excellent but redundant features.

Wrapper methods: fairer, more expensive​

A wrapper evaluates subsets by actually training the model. The most used is recursive feature elimination: train, remove the least important feature, repeat.

from sklearn.feature_selection import RFECV

RFECV(estimator=model, step=1, cv=5, scoring="roc_auc")

The RFECV variant additionally determines the optimal number of features by cross-validation, avoiding an arbitrary choice. The approach accounts for interactions and matches the intended model; its computational cost, however, bears no comparison with a filter's.

Embedded methods: the model selects by itself​

Some models perform selection during training. Lasso drives the least useful coefficients to zero, as seen in course 04: the surviving features are the selected features. Tree-based models provide directly usable feature importances.

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LassoCV

SelectFromModel(LassoCV(cv=5))                    # features with non-zero coefficient
SelectFromModel(tree_model, threshold="median")   # above median importance

This is the best cost-benefit ratio: selection is consistent with the final model and requires a single training run.

Permutation importance: the most reliable measure​

Tree-derived importances have two known biases, flagged in course 04: they favor features with many levels, and they split arbitrarily among correlated features.

Permutation importance sidesteps these biases with a direct idea: randomly shuffle a feature's values and measure the performance drop. If shuffling a feature changes nothing, it was not being used.

from sklearn.inspection import permutation_importance
r = permutation_importance(model, X_val, y_val, n_repeats=10, scoring="roc_auc")

Two decisive qualities: the measure works on any model, and it is computed on validation data, so it measures real usefulness in generalization rather than contribution to the fit. A negative importance flags a feature that hurts — remove it without hesitation.

A pragmatic approach

What works best in practice combines the approaches: clean near-constant and near-duplicate features, trim by mutual information if the column count is high, then refine by permutation importance on the chosen model. And set the threshold by cross-validated performance, not by an arbitrary rule: keep the smallest set whose performance stays within the noise of the maximum. The operational gain then far exceeds the tenth of a point sacrificed.

Summary​

  • You reduce for performance, speed, interpretability and above all the operating cost of every feature in production.
  • Filters (mutual information preferably) are fast but judge each feature in isolation, missing interactions.
  • Wrappers (recursive elimination, RFECV) account for interactions at a heavy computational cost; embedded methods (Lasso, tree importances) offer the best compromise.
  • Permutation importance is the most reliable measure: model-agnostic, computed on validation, and a negative value flags a harmful feature.

Next module: scikit-learn pipelines, which make this entire course reproducible and immune to leakage.