Module 1 — Framing a recommendation problem

Almost every failed recommender in production died at this module — long before it was trained. A team spent months squeezing tenths of RMSE out of a rating predictor while their real business KPI was course completions per learner per quarter. The model was accurate. The recommendations were useless. This module is dedicated to not making that mistake.

Our thread throughout the course is an online learning catalog modeled on InSkillML: about 500 premium and discovery courses, tens of thousands of learners, and two kinds of signals per user-course pair — an explicit star rating (1 to 5) when the learner leaves one, and a stream of implicit events (viewed, enrolled, minutes watched, completed, certificate obtained). The team must produce, every morning, a personalized list of five course recommendations shown on the home page.

Prediction is not the goal, ranking is

The classical formulation, popularized by the Netflix Prize, treats recommendation as predicting an unseen rating. Given a matrix $R$ where $R_{ui}$ is the rating user $u$ gave item $i$ (missing when unknown), find a model $\hat{R}_{ui}$ that minimizes the squared error on the observed entries. It is neat, it is mathematically clean, and it is almost always a mis-framing.

The reason is that our home page never asks "will the user rate this course 4.7 stars?". It asks "which five courses should I put in front of this learner right now, from the 500 available, so that at least one is clicked and at least one is completed?". Two very different questions.

Concretely, imagine a model whose RMSE on ratings is excellent. Its five top predictions for a given user are five courses the learner has already completed, because the model learned that its most confident predictions live where it has the most data — that is, on the courses the user already knows. RMSE is unaware of that; ranking metrics are not. A model with a slightly worse RMSE but a training loss that penalizes missing the top-k list will produce a list one order of magnitude more useful.

The right framing is therefore: produce a ranked list of items for each user, and measure it with ranking metrics (recall@k, NDCG@k, coverage) instead of point-prediction errors. Module 8 goes into these metrics in detail. Everything else — collaborative filtering, factorization, two-tower — is a means to that end.

Two kinds of feedback, and why implicit dominates

Explicit feedback is a deliberate signal from the user: a star rating, a thumbs-up, a written review. It is unambiguous in intent (the learner meant to say something), but it is rare. On our platform, fewer than 4 % of enrollments end with a rating. The 96 % of learners who never leave one are not a random subset — they include the disengaged, the completers who forget to click, and every non-native speaker of the UI language. Selection bias runs deep.

Implicit feedback is any behavioral trace: a page view, an enrollment, a minute-watched counter, a certificate delivery, a bookmark. It is available on every learner (roughly a hundred times more data per user), but it is ambiguous: an enrollment followed by zero minutes watched may be a mis-click, a curiosity spike, or a colleague looking over the shoulder. The absence of an event is even trickier — did the user not want the course, or not see it? Module 9 is devoted entirely to reading implicit feedback without the models it fools.

For our catalog we will use both, and the design rule is simple: train mostly on implicit feedback (there is orders of magnitude more of it), calibrate on explicit ratings when we have them (a five-star rating is a stronger positive than a 30-second view), and treat missing entries with much more care than a rating regressor would.

The rating matrix, and why it is almost entirely empty

Write $|U|$ for the number of users and $|I|$ for the number of items. The user-item matrix $R \in \mathbb{R}^{|U| \times |I|}$ has $|U| \times |I|$ cells. On our platform, with 40 000 learners and 500 courses, that is 20 million cells. Observed cells: about 90 000 explicit ratings, plus 3 million implicit events. Even the implicit matrix is filled at 15 % — the explicit one, at 0.45 %.

That is the defining constraint of recommendation, and it deserves a name: sparsity. Two consequences follow, both of which will surface in later modules.

First, most methods you know from tabular ML break. You cannot compute a mean over a row when 99.5 % of the row is missing; you cannot impute with zeros without lying (a zero here means "unrated", not "hated"); you cannot run a random forest on a matrix that is 99.5 % NaN. Module 3 will replace the missing-value problem with factorization on observed entries only.

Second, most user-user or item-item similarity computations rest on very few co-rated items. Two users who both rated exactly three of the same courses give a "correlation" that is essentially noise. Module 2 addresses this with shrinkage; module 4 sidesteps it by using content instead.

Sparsity, quickly, on your own catalog

import numpy as np
import pandas as pd
from scipy.sparse import coo_matrix

ratings = pd.read_parquet("catalog/ratings.parquet")   # user_id, item_id, rating, ts
n_users = ratings["user_id"].nunique()
n_items = ratings["item_id"].nunique()

uidx = {u: i for i, u in enumerate(ratings["user_id"].unique())}
iidx = {c: i for i, c in enumerate(ratings["item_id"].unique())}

rows = ratings["user_id"].map(uidx).to_numpy()
cols = ratings["item_id"].map(iidx).to_numpy()
data = ratings["rating"].astype(np.float32).to_numpy()

R = coo_matrix((data, (rows, cols)), shape=(n_users, n_items)).tocsr()
density = R.nnz / (n_users * n_items)

print(f"users={n_users}, items={n_items}, ratings={R.nnz}, density={density:.4%}")
print("median ratings per user:", int(np.median(np.diff(R.indptr))))
print("median ratings per item:", int(np.median(np.diff(R.tocsc().indptr))))

On our catalog this prints density=0.4527%, median ratings per user: 2, median ratings per item: 91. The median user has rated two courses. Half the users have less than that. Any method that requires a good estimate per user will fail on that half.

The business objective is a KPI, not a loss

Here is the last framing step, the one teams routinely skip. Before picking a loss, name the KPI the recommender is supposed to move. For our platform, three candidates are on the table:

• Click-through rate on the recommended list — cheap, fast, but easily gamed by showing whatever is most clickable regardless of quality.
• Course completion rate among enrollments driven by the recommendations — the honest signal of usefulness, delayed by weeks and much noisier.
• Learner retention at 30 days — even more useful, even more delayed, and confounded by everything else the platform does.

The framing decision is a trade-off between what the model can optimize offline (a proxy) and what actually matters online (the KPI). Any offline experiment in this course will report a proxy metric — recall@10, NDCG@10 — and every claim will be honest about the gap. Module 10 discusses how an A/B test on retention closes that gap, and why it never disappears.

A metric on the wrong task is worse than no metric

An RMSE of 0.87 on held-out ratings is a real number. It is also a red herring if you never show your users an unseen rating. Optimize what you deliver — a ranked list at position 1 through 10 — with a loss that reflects that. Otherwise a "better" model is genuinely worse for the business.

Summary

• Recommendation is ranking, not rating prediction: RMSE optimizes the wrong page and often surfaces items the user already consumed.
• Explicit feedback is rare and biased; implicit feedback is abundant and ambiguous; the pipeline uses both, and treats their absences differently.
• The user-item matrix is extremely sparse (typically well below 1 %); this defines what methods survive and forces every module to answer "what do we do about missing entries?".
• The offline loss is only a proxy; the framing step names the actual KPI (completion, retention) and reports the proxy-KPI gap honestly.

Next module: the oldest, most intuitive family of recommenders — collaborative filtering by neighborhood — and its practical limits on our sparse catalog.