Module 8 — Metrics: recall, NDCG, coverage, diversity

You cannot improve what you do not measure, and in recommendation the wrong measurement is worse than none. This module builds the four families of metrics that matter for a top-$k$ list, computes NDCG by hand once because everyone should have done it, and closes with the metrics you should log alongside relevance to avoid shipping a filter bubble that scores well.

Precision and recall at k

For a given user, the recommender returns a ranked list of $k$ items $\hat{L}_u = [i_1, i_2, \dots, i_k]$. The ground truth is the set of items the user interacted with in the test window, call it $L_u^\star$.

Precision@k is the fraction of the $k$ recommended items that are relevant:

$$\text{Precision@}k = \frac{|\hat{L}_u \cap L_u^\star|}{k}.$$

Recall@k is the fraction of the relevant items that made it into the top-$k$ list:

$$\text{Recall@}k = \frac{|\hat{L}_u \cap L_u^\star|}{|L_u^\star|}.$$

Both are averaged over users. Precision penalizes irrelevant recommendations shown to the user; recall penalizes missing the ones that would have mattered. Which one you optimize depends on the surface: for a top-3 carousel where slot space is scarce, precision matters most; for a "you might also like" list of 20 items, recall wins.

On our online learning catalog we default to recall@10, because ten is the size of the home-page list and because our users tend to click at most one of them per session, so precision has almost no dynamic range.

The one flaw both share

Precision and recall at $k$ ignore order within the top-$k$. A recommender that puts the relevant item in slot 1 gets the same score as one that puts it in slot 10, provided both keep it inside the top-10. In practice this is wrong: users scan lists top-down, click rates drop sharply with position, and slot 1 is worth several times what slot 10 is.

That is why we need NDCG.

NDCG, computed by hand once

Discounted Cumulative Gain at $k$ discounts each position by a logarithm, so higher slots count more:

$$\text{DCG@}k = \sum_{p=1}^{k} \frac{\text{rel}_p}{\log_2(p + 1)},$$

where $\text{rel}_p \in \{0, 1\}$ marks whether the item at position $p$ is relevant. To make it comparable across users, divide by the DCG of the ideal ordering:

$$\text{IDCG@}k = \sum_{p=1}^{\min(k, |L_u^\star|)} \frac{1}{\log_2(p + 1)}, \qquad \text{NDCG@}k = \frac{\text{DCG@}k}{\text{IDCG@}k}.$$

Do this one time on paper. Suppose $k = 5$ and the user has two relevant items in the test set. The recommender returns, in order, $[i_a, i_b, i_c, i_d, i_e]$, and only $i_b$ and $i_d$ are in $L_u^\star$. Then:

$$\text{DCG@5} = \frac{0}{\log_2 2} + \frac{1}{\log_2 3} + \frac{0}{\log_2 4} + \frac{1}{\log_2 5} + \frac{0}{\log_2 6} = 0.631 + 0.431 = 1.062.$$

The best possible ordering with two relevants would be to put both at positions 1 and 2:

$$\text{IDCG@5} = \frac{1}{\log_2 2} + \frac{1}{\log_2 3} = 1.000 + 0.631 = 1.631.$$

So $\text{NDCG@5} = 1.062 / 1.631 = 0.651$.

If the recommender had put the two relevants at positions 1 and 2, DCG would equal IDCG and NDCG would be 1.0. Same recall (both cases recover 2 of 2 relevants), very different NDCG.

The Python that ships

import numpy as np

def dcg_at_k(rels: np.ndarray, k: int) -> float:
    r = rels[:k].astype(float)
    positions = np.arange(1, len(r) + 1)
    return float(np.sum(r / np.log2(positions + 1)))

def ndcg_at_k(recommended: list[int], relevant: set[int], k: int = 10) -> float:
    rels = np.array([1 if i in relevant else 0 for i in recommended])
    dcg = dcg_at_k(rels, k)
    ideal = np.sort(rels)[::-1]
    idcg = dcg_at_k(ideal, k)
    return dcg / idcg if idcg > 0 else 0.0

def mean_ndcg(recommender, test_by_user: dict[int, set[int]], k: int = 10) -> float:
    scores = [ndcg_at_k(recommender(u, k), rel, k) for u, rel in test_by_user.items() if rel]
    return float(np.mean(scores))

The implementation deserves one comment. When a user has no items in the test window (|L_u^\star| = 0), NDCG is undefined; skip those users, do not return zero. Silently mixing zero-relevant users into the average is a common source of "our NDCG dropped after the split change" surprises that are entirely due to bookkeeping.

The library torchmetrics and pytrec_eval provide production-tested implementations; use them for anything you serve, but write the hand version once so that you own the definition.

Coverage: what fraction of the catalog you show

Catalog coverage at $k$ is the fraction of items that appear in at least one user's top-$k$ over the test period:

$$\text{Coverage@}k = \frac{|\{i : \exists u,\, i \in \hat{L}_u^{(k)}\}|}{|I|}.$$

A recommender that always shows the same 40 courses to everyone has coverage 40 / 500 = 8 %. It might have excellent recall@10 by matching the head of the catalog for everyone, and it would be quietly killing the long tail of your business.

There is a variant, user coverage, that measures the fraction of users for whom the recommender produces at least one relevant suggestion. It catches the opposite problem: a recommender that ignores your low-activity users entirely.

Track both. On our platform we hold coverage above 60 % — below that, the catalog editor calls a meeting.

Diversity within a list

Intra-list diversity measures how different the items inside a single user's top-$k$ list are from each other:

$$\text{ILD@}k = \frac{2}{k(k-1)} \sum_{p<q} \left(1 - \text{sim}(i_p, i_q)\right),$$

using item content similarity (module 4) as the sim measure. A list of ten Docker courses has ILD near zero; a list of ten courses across different topics has ILD near one.

Diversity matters because a top-10 of ten near-identical items reduces to essentially one recommendation. Users click at most one; showing them ten copies of the same idea wastes nine slots.

Maximal Marginal Relevance re-ranks the top-200 candidates to trade a bit of relevance for a bit of diversity:

$$\text{MMR}(i) = \lambda\, s(u, i) - (1 - \lambda)\, \max_{j \in S} \text{sim}(i, j),$$

where $S$ is the set of items already selected for the list and $\lambda \in [0, 1]$ controls the trade-off. $\lambda = 0.7$ is a reasonable default — mostly relevance-driven, with a light diversity nudge.

Novelty

Novelty measures how surprising a recommendation is, typically by the log-inverse of the item's popularity:

$$\text{Novelty}(i) = -\log_2 \Pr(i),$$

averaged over the recommended list. A recommender that always shows the top-100 most popular items has zero novelty. On our platform, novelty is a leading indicator for medium-term retention: users who see a mix of high-novelty items in their first week come back more often at day 30.

Recall, coverage, diversity and novelty are the four numbers we log in every offline evaluation. Any change that improves recall while degrading two of the others is a change we scrutinize before shipping.

RMSE for a ranked list is misleading

Module 1 warned that RMSE optimizes the wrong page; module 3 emphasized that regularization tuned on RMSE is not the regularization for a ranker. Module 8 closes the loop with a concrete example.

Consider two models on the same test set:

• Model A: RMSE 0.87, recall@10 0.12.
• Model B: RMSE 0.94, recall@10 0.24.

Model A wins on RMSE by 8 %; Model B wins on recall by a factor of two. If your product is a top-10 list, ship B. If you also report NDCG@10 to your stakeholders, the story is even clearer. RMSE reports a fit; it does not report a page.

The general rule: evaluate the model on the metric of the surface it feeds. A rating widget takes RMSE; a top-10 list takes NDCG@10; a next-item recommender takes hit rate at 1; a search page takes MRR. Never carry a metric across surfaces.

A great single number hides the failure of another

Aggregating everything into one metric is a management convenience, not a modeling one. Any change that materially affects a recommender should be reviewed on four numbers: relevance (recall or NDCG), coverage, diversity and novelty. A win on one that costs two of the others is a regression pretending to be an improvement.

Summary

• Precision@k and recall@k measure how many relevant items make the top-$k$, ignoring order; NDCG@k adds the position discount that reflects how users actually scan.
• Compute NDCG by hand once with the $1 / \log_2(p+1)$ discount; the definition owns you until you do.
• Catalog coverage, intra-list diversity and novelty are the health metrics that keep a recommender from shipping a filter bubble; log all four, not just relevance.
• Never optimize a ranked-list surface with RMSE; a rating error minimum and a ranking-metric minimum sit at different hyperparameter settings, sometimes an order of magnitude apart.

Next module: the implicit feedback the modern web actually produces — clicks, dwell times, positions — and the biases that turn them into a distorted training signal.