Module 10 — Project: an evaluated recommendation engine

Nine modules of pieces; one module to fit them together into something you could actually ship. We build the full pipeline for the online learning catalog of module 1: temporal split, four models compared on the same held-out window, a single metrics table, an online A/B test plan, and — mandatory — a short section on what this engine cannot do.

The temporal split, done properly

Every trap in offline evaluation of a recommender is a variant of "training data leaked into test data through time". Getting the split right is worth more than any modeling improvement in the rest of this section.

The rule: choose a cutoff timestamp $t^\star$; every interaction with $\text{ts} < t^\star$ is training, every interaction with $\text{ts} \geq t^\star$ is test. Do not split by user; do not split randomly; do not shuffle within a user. This produces two important properties:

• Every model sees only what would have been available at time $t^\star$ — no future course descriptions, no future ratings, no future interactions.
• Users are naturally partitioned into "warm" (had activity before $t^\star$) and "cold" (their first interaction is after $t^\star$), so the cold-start metrics of module 7 fall out of the same split.

import pandas as pd

events = pd.read_parquet("catalog/events.parquet")   # user_id, item_id, ts, minutes
items = pd.read_parquet("catalog/items.parquet")     # item_id, title, description, tags, added_at

cutoff = pd.Timestamp("2026-06-01")
train = events[events["ts"] < cutoff].copy()
test = events[events["ts"] >= cutoff].copy()

# Cold-item flag from item creation, cold-user flag from training activity
new_items = set(items.loc[items["added_at"] >= cutoff - pd.Timedelta(days=30), "item_id"])
train_users = set(train["user_id"].unique())

One more discipline: when we compute item content embeddings (module 4), we do so on the descriptions available at time $t^\star$. Retraining on later descriptions is a subtle leak that inflates content-based scores because course descriptions get edited as they mature.

Four models on the same table

To make the comparison honest, all four models are trained on the same train and evaluated on the same test, with the same exclusion rule (a recommendation of an item the user already had in train is filtered before scoring).

• A — Item-based CF with shrinkage (module 2), $\lambda = 25$, top-50 neighbors per item.
• B — Matrix factorization (module 3), 64 factors, ALS on implicit, confidence per module 9.
• C — Feature-enriched LightFM (module 5), 64 components, WARP loss, item tags as features.
• D — Two-tower (module 6), 64-d towers, item side vector = 768-d description embedding plus one-hot difficulty and language.

The training call for each is a few lines; the evaluation call is a single function.

import numpy as np

def evaluate(model_predict, test_by_user: dict[int, set[int]], seen_by_user: dict[int, set[int]], k=10):
    from module_08_metrics import ndcg_at_k
    recalls, ndcgs = [], []
    for u, rel in test_by_user.items():
        if not rel: continue
        candidates = [i for i in model_predict(u, top=200) if i not in seen_by_user.get(u, set())]
        reco = candidates[:k]
        hits = set(reco) & rel
        recalls.append(len(hits) / len(rel))
        ndcgs.append(ndcg_at_k(reco, rel, k))
    return float(np.mean(recalls)), float(np.mean(ndcgs))

test_by_user = test.groupby("user_id")["item_id"].apply(set).to_dict()
seen_by_user = train.groupby("user_id")["item_id"].apply(set).to_dict()

Running the four models on our catalog with the same split produces the following table. Absolute numbers depend on your data; the ordering is stable across everything we have tried.

Model	recall@10	NDCG@10	Coverage@10	ILD@10	Cold-item recall
A. Item-CF + shrinkage	0.127	0.081	0.42	0.31	0.00
B. ALS matrix factorization	0.184	0.121	0.51	0.36	0.00
C. LightFM WARP + tags	0.211	0.144	0.63	0.44	0.09
D. Two-tower + description embeddings	0.239	0.165	0.68	0.47	0.18

Three observations. Model D wins on every metric, at higher training and serving cost. The cold-item column is the true test: A and B are structurally zero because they have no way to score an item that did not appear in train. Coverage rises with the models that can consume features, because those models spread their scores over more of the catalog.

The best offline model is not necessarily the best online model

The table above is offline. What it does not measure:

• Novelty degradation over time: model D at $t^\star$ might do well, but if the same model runs unchanged for six months it will drift as the catalog changes.
• Interaction with the UI: the home page has ten slots but also a "continue learning" section, a "popular this week" carousel, and a search bar. A model that overlaps heavily with those other sections wastes slots.
• Feedback loop under the new model: model D might recommend better items today, but if it consistently under-recommends a category, that category collects no clicks and quietly disappears from the training signal in three months.

The bridge from offline to online is an A/B test. Choose a fraction of traffic — 10 % is a reasonable start on our size platform — and route it to model D while the rest continues to see model C. Measure the KPIs we named in module 1 (clicks, enrollments, completions, retention at 30 days) and let the test run for at least two full weekly cycles (14 days minimum on our catalog).

Three A/B pitfalls that show up on every recommender:

• Novelty effect: users click more on anything new, so the first 3–5 days of an A/B typically overstate the winner. Trust the second week.
• Segment interaction: a model that lifts overall completions may hurt cold-user retention. Slice A/B results by segment; a global lift that comes from a segment loss is a bad ship.
• Winner's curse: the winning arm looks slightly better than it really is because you selected on the outcome. Do a validation A/B — the ship candidate against itself, on independent traffic — before scaling to 100 %.

Serving pattern

A shipping pipeline for model D looks like this:

1. Nightly training on the last 90 days of interactions, on a snapshot pinned to a specific timestamp.
2. Item vector indexing: run the item tower on every catalog item; write the vectors to a FAISS index; ship the index behind a small service.
3. User vector at request time: on each home-page load, look up the user's precomputed features, run the user tower once (a few milliseconds), query the index for the top-200 candidates.
4. Ranker (module 5 or a small gradient boosting model on cross-features): score the 200 candidates and produce the top-10.
5. Business rules layer: enforce cold-item ceiling, filter out items outside the user's language, cap consecutive items from the same topic, reserve one slot for exploration.

The business rules layer catches everything that is easier to write down than to model: "do not recommend a course in a language the user does not speak", "if the user has an active enrollment in the same series, prefer the next course in the series". These are usually 5 % of the code and 50 % of the incidents avoided.

Limits, honestly stated

Every recommender ships with limits. Naming them in the design doc is the difference between a healthy production system and a firefight.

• We cannot recommend to a user we have never seen before their first click. For the first request of a new user, we fall back to segmented popularity (module 7); this is measured and known-imperfect.
• We cannot detect a fundamental change in a user's interests. A learner who switches from Python to cybersecurity will get Python recommendations for weeks until the model catches up. A "reset my interests" button in the UI is worth more than any model tweak here.
• We do not model diversity of exposure across users. Two users with similar profiles receive very similar lists; nothing stops the pipeline from creating a small number of tightly overlapping recommendation clusters. A team should periodically audit list overlap.
• The offline metrics predict online metrics imperfectly. The gap between recall@10 offline and enrollments online is not a constant; it drifts with the UI, the market, and the season.

Every one of these is a follow-up ticket, not a blocker. Shipping without naming them is a blocker.

Summary

• The temporal split at a cutoff $t^\star$ is the only honest offline evaluation; every content embedding, every feature and every model must respect it.
• Compare all models on the same table with recall@10, NDCG@10, coverage, diversity and cold-item recall — one row per model, one column per metric; ordering is stable across catalogs.
• The offline table is a starting point; a two-week A/B test with segment slicing is what tells you which model to ship, with the winner's curse and novelty effect explicitly checked.
• Every recommender ships with known limits — new users, drift, exposure diversity, offline-online gap. Name them; the alternative is discovering them in an incident.

Next module: the course recap and the final 40-question exam, with the decision grid you should carry into your first real project.