Skip to main content

Module 3 — Scaling: normalization and standardization

The two previous courses repeatedly said standardization was indispensable. This module finally explains why, for which models, and above all which ones do perfectly well without it. Confusing the two families wastes time in one direction, and performance in the other.

Which models require it, which do not care​

The rule is clear once stated correctly: scaling matters for any model that computes distances, dot products, or optimizes by gradient descent.

Scale-sensitiveScale-indifferent
k-nearest neighbors, SVMdecision trees
k-means, DBSCAN, PCArandom forests
regularized regressions (Ridge, Lasso)gradient boosting
neural networks—

The right-hand column has a simple explanation: a tree never compares two variables with each other. It asks questions such as "income > 30,000?", on one variable at a time. Multiply every income by a thousand and the tree will find exactly the same threshold, a thousand times larger, with the same partition. No scaling is needed for tree-based methods, and this is one reason for their popularity on heterogeneous tabular data.

One case deserves mention: regularization. Ridge and Lasso penalize coefficient size, yet a coefficient depends on the unit of its variable. Without scaling, the penalty arbitrarily hits small-magnitude variables harder — tuning becomes incoherent.

The three transformations to know​

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

StandardScaler()   # (x - mean) / std   -> mean 0, std 1
MinMaxScaler()     # (x - min) / (max - min)  -> bounded in [0, 1]
RobustScaler()     # (x - median) / IQR  -> insensitive to outliers

Standardization is the default choice. It centers and scales without bounding, and assumes no known extreme values. It is what PCA and regularized models expect.

Min-max normalization bounds within a fixed interval, useful when a precise range is required — neural network inputs, image processing. Its weakness is stark: a single outlier sets the maximum and crushes all other values into a tiny portion of the interval.

Robust scaling uses median and interquartile range. It is the right reflex in the presence of outliers you neither want to remove nor let dominate, for the same reason as in the previous module.

Scaling does not fix the shape of the distribution​

A frequently misunderstood point, and a source of disappointment. Standardization changes the scale, not the shape: a strongly skewed distribution remains skewed after standardization, with the same tail.

Yet skewness genuinely hampers linear models, which assume regular relationships. To act on shape you need other transformations:

  • logarithm (np.log1p, which handles zero) for long-tailed distributions — amounts, populations, waiting times. By far the most useful;
  • square root, a similar but gentler effect;
  • Box-Cox or Yeo-Johnson (PowerTransformer), which automatically search for the transformation bringing the distribution closest to a normal;
  • discretization into bins (KBinsDiscretizer), which gives up granularity to capture threshold effects.

The logarithm deserves an extra word: on an amount variable, it turns multiplicative gaps into additive ones. Going from 10 to 100 euros becomes equivalent to going from 100 to 1,000 — which often matches the economic reality of the phenomenon far better than a raw difference.

Fit on training, transform everywhere

The same trap as in module 2, and it applies to every transformation in this course. Mean, standard deviation, minimum, maximum, median: these statistics are computed on training data only (fit), then applied to the test set (transform). A fit_transform on the full dataset is silent data leakage — the score rises, production disappoints.

Summary​

  • Scaling matters for models based on distances, dot products or gradient descent; tree-based methods do entirely without it.
  • Standardization by default, min-max when a bounded range is required, robust in the presence of outliers.
  • Scaling does not change the shape of the distribution: skewness is treated by logarithm, Yeo-Johnson or discretization.
  • All transformation statistics are fit on training data alone, then applied to the test set.

Next module: encoding categorical variables, where the choice of method depends directly on cardinality.