Skip to main content

Module 7 — Features from aggregations and time windows

Previous modules transformed one row at a time. Here, information is built by looking at several rows together: a customer's history, a category average, the trend of the last seven days. This is where feature engineering produces its largest gains — and where data leakage lurks at every step.

Aggregating by group: summarizing a history​

The typical case: a transactions table, several rows per customer, and a model that predicts at customer level. The history must be summarized into features.

agg = df.groupby("customer_id").agg(
    n_purchases    = ("amount", "count"),
    amount_mean    = ("amount", "mean"),
    amount_median  = ("amount", "median"),
    amount_max     = ("amount", "max"),
    amount_std     = ("amount", "std"),
    n_categories   = ("category", "nunique"),
)

Each aggregation function captures a distinct aspect of behavior. Count measures the intensity of the relationship, the mean the usual level, the median that same level while resisting extremes, the maximum peak capacity, the standard deviation regularity — a customer with a stable basket does not behave like an erratic customer with the same mean. The number of distinct values measures diversity, often highly predictive of loyalty.

Deviation from habit: the feature that changes everything​

Here is the type of feature mentioned in module 1, and it deserves to be built systematically. A 500-euro transaction means nothing in the absolute: it is mundane for a customer who spends 600 euros on average, and highly suspicious for a customer usually at 30 euros.

The information is therefore not the amount, but the gap between the amount and the habit:

df["customer_mean"] = df.groupby("customer_id")["amount"].transform("mean")
df["customer_std"]  = df.groupby("customer_id")["amount"].transform("std")
df["relative_gap"]  = (df["amount"] - df["customer_mean"]) / df["customer_std"]

That last feature is a z-score computed per customer rather than over the population. This is a pattern of reasoning to reuse widely: relate a value to its relevant reference — the customer's history, the average of its product category, the seasonal norm. Many performance gains come from there, not from a more sophisticated algorithm.

Note that transform is preferable to agg here: it returns one value per original row and therefore joins naturally, without a manual merge.

Rolling windows and lags: the golden rule​

On time-ordered data, you want to summarize the recent past: mean of the last seven days, sum of the past month, trend. These features are powerful and represent the main source of leakage in this course.

The rule is absolute: a window must contain strictly past data only. A centered moving average, or a window including the current row, uses the very value you are trying to predict.

df = df.sort_values(["customer_id", "date"])

# Previous value: the past, unambiguously
df["prev_amount"] = df.groupby("customer_id")["amount"].shift(1)

# Mean of the 7 previous rows: shift(1) BEFORE rolling
df["mean_7"] = (
    df.groupby("customer_id")["amount"]
      .transform(lambda s: s.shift(1).rolling(7, min_periods=1).mean())
)

The shift(1) placed before rolling is the detail that decides the validity of the entire model. Without it, the window includes the current row: the feature contains part of the target, the training score becomes spectacular, and production collapses. This is the most frequent error on temporal data.

Once these foundations are laid, derived features follow: the ratio between the 7-day mean and the 30-day mean measures acceleration; the difference from the previous value measures change; the number of days since the last event measures recency at row level.

Aggregating by category rather than by individual​

Aggregation is not limited to individuals. Summarizing by product category, by region, by day of week provides a reference against which to relate each observation: is this product expensive for its category? Does this store outperform its region?

One caution, though: aggregating the target by category is the target encoding of module 4, with the same requirement of smoothing and out-of-fold computation. Aggregating variables other than the target does not raise that problem.

Three checks before validating an aggregated feature

First, sort order: without sort_values on the date, shift and rolling silently produce nonsense. Second, the group: forgetting the groupby mixes the histories of several customers. Third, availability: will this aggregation be computable in production, at prediction time, with only the data already known? A feature requiring a full table recomputed nightly does not carry the same cost as a simple shift.

Summary​

  • Group aggregation summarizes a history; count, mean, median, maximum, standard deviation and number of distinct values each capture an aspect of behavior.
  • Deviation from habit (per-individual z-score) is often more predictive than the raw value: relating a value to its relevant reference is a reflex to generalize.
  • A rolling window must contain strictly past data: shift(1) before rolling, on pain of massive leakage.
  • Aggregating by category provides useful references, but aggregating the target amounts to target encoding and its precautions.

Next module: data leakage, its most frequent forms and the signals that should raise the alarm.