Module 3 — Matrix factorization and SVD

Neighborhood methods store a similarity table proportional in size to the number of items squared. Matrix factorization replaces that with something much cheaper and much more expressive: a small number of latent factors per user and per item. A dot product produces the score. In two hundred numbers per user, you capture what a several-thousand-entry similarity row could not.

This module gives the objective, the two optimizers you must know, the two regularizations you must not skip, and a working implementation on the catalog of module 1.

Why "latent factors" and not "PCA on the ratings"

The name is unfortunate. In introductory texts it looks like SVD applied to the rating matrix, which invites you to run numpy.linalg.svd(R). Do not. The rating matrix has 99.5 % missing entries — filling them with zeros before SVD tells the model that unrated is disliked, which is exactly what module 1 warned against, and produces a decomposition that fits the imputation, not the users.

The right formulation is a factorization on observed entries only. We look for two low-rank matrices $P \in \mathbb{R}^{|U| \times k}$ and $Q \in \mathbb{R}^{|I| \times k}$ such that, on the cells $(u, i)$ where $R_{ui}$ is observed,

$$R_{ui} \approx b_g + b_u + b_i + \mathbf{p}_u^{\top} \mathbf{q}_i.$$

Each row $\mathbf{p}_u$ is a $k$-dimensional vector — the user factors — and each row $\mathbf{q}_i$ the item factors. Their inner product is the model's prediction of user $u$'s taste for item $i$. The values $k$ typically live between 32 and 200. The three bias terms are the second half of the story, and skipping them is the most common mistake I see in first implementations.

The three biases you cannot skip

$b_g$ is the global mean — around 3.7 stars on our catalog. $b_u$ is a per-user offset — some users rate 4.5 on average, others 2.5, and this alone explains a large chunk of the variance without any personalization. $b_i$ is a per-item offset — a fundamentals-of-Python course averages 4.6 stars, an advanced-topology course 3.2, before we ever look at who rated it.

Removing the biases makes the model use its factors to explain that user 42 is generally happy, instead of using them to describe what user 42 likes. Two orders of magnitude of quality are on the table. Always fit the biases.

The objective, and the two optimizers

The training objective is a regularized squared error on observed entries:

$$\mathcal{L} = \sum_{(u,i) \in \Omega} \left(R_{ui} - b_g - b_u - b_i - \mathbf{p}_u^{\top}\mathbf{q}_i\right)^2 + \lambda\left(\|\mathbf{p}_u\|^2 + \|\mathbf{q}_i\|^2 + b_u^2 + b_i^2\right),$$

where $\Omega$ denotes the set of observed cells.

Stochastic gradient descent picks one observed triplet at a time, computes the error $e_{ui}$, and updates every parameter it touches:

$$\mathbf{p}_u \leftarrow \mathbf{p}_u + \eta\,(e_{ui}\,\mathbf{q}_i - \lambda\,\mathbf{p}_u), \qquad \mathbf{q}_i \leftarrow \mathbf{q}_i + \eta\,(e_{ui}\,\mathbf{p}_u - \lambda\,\mathbf{q}_i).$$

Simple, works with a mini-batch, easy to add features (temporal biases, side information). The learning rate matters; start with 0.005 and halve on plateau.

Alternating least squares fixes $Q$ and solves for $P$ in closed form (each row of $P$ becomes a small ridge regression), then fixes $P$ and solves for $Q$. Each subproblem is convex; ten iterations usually suffice. ALS parallelizes beautifully because the users are independent given $Q$, and this is what production systems on Spark use. The library implicit uses ALS for its implicit-feedback variant, which we return to in module 9.

Which to pick? SGD when you want easy extensions, ALS when you want speed on a cluster. On a single machine and 500 courses, both converge in seconds.

Regularization is not optional

With $|U| \cdot k + |I| \cdot k$ parameters — for us, about 8 million with $k = 200$ — the model has vastly more capacity than the 90 000 observed ratings. Without $\lambda$, it overfits catastrophically: training RMSE drops to 0.1 while the held-out RMSE rises past 1.5.

A practical rule: start with $\lambda = 0.1$, use $k = 64$, evaluate on a held-out user-time split (module 9), then tune both. Two things to watch:

• $\lambda$ interacts strongly with $k$. Increasing $k$ demands more regularization to keep held-out error stable; do not tune them independently.
• Very common to see a training RMSE that keeps dropping while the held-out one has stopped moving — that is the fingerprint of a regularization that is too weak.

A working implementation on the catalog

We use the implicit library for ALS on implicit data (enrollments, minutes watched) and surprise for classical explicit-rating SVD. Both are trivial to install and both scale to our size.

import numpy as np
import pandas as pd
from scipy.sparse import csr_matrix
import implicit

events = pd.read_parquet("catalog/events.parquet")   # user_id, item_id, minutes
# Confidence weight: minutes watched, capped
events["conf"] = 1.0 + np.log1p(events["minutes"].clip(0, 240))

u = events["user_id"].astype("category")
i = events["item_id"].astype("category")
Ci = csr_matrix((events["conf"].values, (u.cat.codes.values, i.cat.codes.values)))

model = implicit.als.AlternatingLeastSquares(
    factors=64, regularization=0.1, iterations=15, use_gpu=False
)
model.fit(Ci)

# Top-10 for user 42 (excluding items already in Ci)
user_row = Ci[42]
recos, scores = model.recommend(42, user_row, N=10, filter_already_liked_items=True)
print(list(zip(u.cat.categories[recos], scores.round(3))))

The use_gpu=False is deliberate — on 500 items and 40 000 users, the CPU is faster than moving data to a GPU. Turn it on when the item count crosses a hundred thousand.

For the explicit case, surprise gives a SVD estimator with the exact objective above. It is a hair slower than a hand-rolled SGD but its cross-validation utilities are worth the seconds.

from surprise import Dataset, Reader, SVD
from surprise.model_selection import cross_validate

ratings = pd.read_parquet("catalog/ratings.parquet")[["user_id", "item_id", "rating"]]
reader = Reader(rating_scale=(1, 5))
data = Dataset.load_from_df(ratings, reader)

algo = SVD(n_factors=64, reg_all=0.1, n_epochs=25, lr_all=0.005)
scores = cross_validate(algo, data, measures=["RMSE", "MAE"], cv=3, verbose=True)

Cross-validation here is user-random by default. That is fine for a first sanity check but almost always over-optimistic; module 9 replaces it with a leave-one-out-per-user-latest-in-time split, which is the honest offline evaluation for a ranking system.

Reading the learned factors

Latent factors are almost always uninterpretable one by one. Their axes are arbitrary and rotationally invariant — no reason the third component would correspond to "difficulty" or "programming vs theory". What is interpretable is item-item cosine similarity in the factor space:

from sklearn.metrics.pairwise import cosine_similarity
Q = model.item_factors                                 # items x k
sim_q = cosine_similarity(Q)
titles = pd.read_parquet("catalog/items.parquet").set_index("item_id")["title"]
anchor = titles.index.get_loc("intro-python")
for j in np.argsort(-sim_q[anchor])[1:6]:
    print(titles.iloc[j], sim_q[anchor, j].round(3))

On our catalog this returns "advanced-python", "pandas-in-practice", "python-for-data", "flask-basics", "python-testing" — a clean neighborhood learned from co-consumption, without ever reading a course description. That is the daily win of factorization.

Regularization tuned on the wrong metric

Tuning $\lambda$ by minimizing RMSE and shipping the model as a ranker is exactly the mis-framing of module 1. If your production KPI is a top-$k$ list, tune $\lambda$ against NDCG@10 or recall@10 (module 8), not against a rating error. Two different sweet spots, sometimes an order of magnitude apart in $\lambda$.

Summary

• Matrix factorization learns latent factors per user and item; a dot product gives the prediction, and the size $k$ trades expressiveness for regularization.
• The three biases (global, per-user, per-item) explain most of the variance and are the single most common omission in first implementations.
• SGD and ALS both solve the objective on observed entries only; ALS scales on a cluster, SGD extends more easily to features.
• Tune the regularization $\lambda$ jointly with $k$ and against the metric you actually deliver (a ranking metric), not against RMSE.

Next module: what to do when a course has zero interactions or a very new user has none — the content-based angle, on descriptions instead of ratings.