Module 6 — Deep recommendation and embeddings

Matrix factorization gives each user and item a dense vector, then scores by dot product. A two-tower model does exactly the same at retrieval time — one embedding per user, one per item, dot product — but computes each embedding through a small neural network that can consume features. This one design decision, popularized by YouTube in 2016 and now standard at every large recommender, is the bridge between classical CF and modern deep systems.

This module builds a small two-tower in PyTorch on our catalog, sets up in-batch negative sampling, and closes with the retrieval-then-ranking split that makes billions-scale recommenders tractable.

Why two towers instead of one big network

A single network taking (user features, item features) and outputting a score is more expressive than a two-tower. It is also almost useless at scale. Consider serving: for each user, you must score every candidate item. With 40 000 users, 500 items and a joint model, that is 20 million forward passes per full pass over the catalog; with a real recommender at 100 million users and 10 million items, it is $10^{15}$ forward passes, which is unaffordable.

A two-tower factorizes the computation. The user tower produces $\mathbf{u} = f_\theta(\text{user features})$ once. The item tower produces $\mathbf{v}_i = g_\phi(\text{item features})$ once per item, cached in an index. Retrieval is $\mathbf{u}^{\top}\mathbf{v}_i$ per candidate, and — thanks to ANN indices — you never actually score every item.

That is the practical reason two towers dominate. The theoretical reason is that the top of every large ranking stack that ships uses the same factorized form; if you want to work at scale, you need to know it.

Structure of the towers

Each tower is a small MLP that consumes features and outputs a fixed-size vector, typically 64 or 128 dimensions. On our catalog:

• User tower input: identity embedding of the user, one-hot country, count of past enrollments (binned), average past rating, total minutes watched (log-scaled).
• Item tower input: identity embedding of the item, sentence embedding of the description (module 4), one-hot difficulty level, one-hot topic, price in USD.

Two design choices matter. First, the two towers must output vectors of the same dimension, so that the dot product is defined. Second, the towers must not share information at training time except through the loss. Any leak (batch normalization statistics computed jointly across both, a shared feature that reveals the label) makes the model impossible to serve because retrieval decouples them.

Negative sampling: the whole training story

The two-tower is trained to make the dot product $\mathbf{u}^{\top}\mathbf{v}_i$ high for user-item pairs the user liked and low for pairs they did not. The problem: we only observe positives (enrollments, completions). No user tells us which of the 499 other courses they did not want. We must invent negatives.

Random negatives — for each positive, sample a random item from the catalog. Cheap, biased toward popular items (they appear often as random samples for users who did not consume them), and produces mediocre rankings.

In-batch negatives — a batch of $B$ positive (user, item) pairs; for user $u$ in the batch, the $B-1$ items paired with other users in the batch serve as negatives. Amortizes computation (one forward pass per tower per batch), yields good models, and is the standard for two-tower training. Its weakness: negatives are biased toward the item distribution of the training data, which favors popular items and produces a popularity bias that module 9 corrects with propensity weighting.

Hard negative mining — after warming up with in-batch negatives, sample negatives that the current model scores high. Typically brings a couple of points of recall@10 at the price of some engineering effort and instability.

For a first ship, in-batch negatives with a moderate batch size (256 to 1024) are the right default.

Loss: sampled softmax

Given a batch $\{(\mathbf{u}_b, \mathbf{v}_b)\}_{b=1}^{B}$, the sampled softmax loss treats each row as a $B$-way classification: for user $u_b$, which item in the batch is the positive?

$$\mathcal{L} = -\frac{1}{B}\sum_{b=1}^{B} \log \frac{\exp(\mathbf{u}_b^{\top}\mathbf{v}_b / \tau)}{\sum_{b'=1}^{B} \exp(\mathbf{u}_b^{\top}\mathbf{v}_{b'} / \tau)}.$$

The temperature $\tau$ controls the sharpness; 0.1 is a sensible starting point, and it interacts with the norm of the embeddings. Two frequent bugs to avoid: forgetting to exclude the positive itself when it appears more than once in a batch (rare but happens), and normalizing embeddings inside the model without adjusting $\tau$ accordingly.

A minimal two-tower in PyTorch

import torch
import torch.nn as nn
import torch.nn.functional as F

class Tower(nn.Module):
    def __init__(self, id_vocab: int, side_dim: int, out_dim: int = 64):
        super().__init__()
        self.id_emb = nn.Embedding(id_vocab, 32)
        self.mlp = nn.Sequential(
            nn.Linear(32 + side_dim, 128), nn.ReLU(),
            nn.Linear(128, out_dim),
        )

    def forward(self, ids, side):
        return F.normalize(self.mlp(torch.cat([self.id_emb(ids), side], dim=-1)), dim=-1)

class TwoTower(nn.Module):
    def __init__(self, n_users, u_side, n_items, i_side, dim=64, tau=0.1):
        super().__init__()
        self.user = Tower(n_users, u_side, dim)
        self.item = Tower(n_items, i_side, dim)
        self.tau = tau

    def forward(self, uid, uside, iid, iside):
        u = self.user(uid, uside)                 # (B, dim)
        v = self.item(iid, iside)                 # (B, dim)
        logits = u @ v.t() / self.tau              # (B, B) in-batch negatives
        labels = torch.arange(u.size(0), device=u.device)
        return F.cross_entropy(logits, labels)

model = TwoTower(n_users=40000, u_side=8, n_items=500, i_side=768 + 10)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)

for uid, uside, iid, iside in loader:              # positive pairs only
    loss = model(uid, uside, iid, iside)
    opt.zero_grad(); loss.backward(); opt.step()

That is not pseudo-code; that trains a working two-tower on our catalog in about ninety seconds on a laptop. The loader yields positive (user, item, features) pairs — one row per interaction that counts as a positive under your rule (enrollment, completion). The iside for an item bundles the 768-d sentence embedding of the description (module 4) with a small one-hot side vector.

Retrieval, then ranking

At serve time, we do not want to compute $\mathbf{u}^{\top}\mathbf{v}_i$ for every item. The pipeline splits into two phases.

Retrieval: at index time, run every item through the item tower, store the vectors in an approximate nearest neighbor index (FAISS, ScaNN, HNSW). At request time, run the user tower once, ask the index for the top-200 items most similar in cosine. Sub-millisecond, catalogs of hundreds of millions.

import faiss
V = model.item.forward(all_item_ids, all_item_side).detach().cpu().numpy().astype("float32")
index = faiss.IndexFlatIP(V.shape[1])              # inner product = cosine on normalized vectors
index.add(V)

Ranking: on the 200 candidates, run a heavier model (cross-features, gradient boosting on user-item interactions, a small ranking network with all the features you could not put in the towers). Because we only ever rank 200 items, we can afford a model that would be untractable over the whole catalog. This is where personalized features that break the two-tower factorization — "how long since the user visited a course in this topic" — belong.

The retrieval-then-ranking split is the single most important architectural pattern in modern recommendation. Every large system uses it. On our catalog it is overkill, but you should build it once now, on 500 items, so that you know what the pieces are before the catalog grows.

Evaluation, quickly

Because our real interest is a ranked list, we evaluate the two-tower with recall@10 and NDCG@10 on a held-out set of interactions, split by time per user (module 9 and 10 detail the split). On our catalog, the numbers move like this:

Model	recall@10	NDCG@10
Item-based CF (module 2)	0.127	0.081
Matrix factorization (module 3)	0.184	0.121
LightFM feature-enriched (module 5)	0.211	0.144
Two-tower with description embeddings	0.239	0.165

The two-tower wins, and its cold-item performance (a subset none of the CF models score above zero on) is substantially better than every predecessor. It also costs the most to serve and to train, and requires an ANN index in production. That trade-off is the point.

A two-tower does not fix a badly framed problem

If your KPI is 30-day retention and you evaluate a two-tower on RMSE, it will lose to a simpler model that happens to optimize the wrong loss less badly. Fix the framing (module 1) and the metrics (module 8) before reaching for depth. Depth amplifies what you optimize; if you optimize the wrong thing, it amplifies the wrong thing.

Summary

• The two-tower model produces a user vector and an item vector independently through two small networks, scored by dot product — the factorization pattern that makes retrieval tractable at scale.
• In-batch negative sampling with a sampled-softmax loss is the standard training recipe; it introduces a popularity bias that module 9 corrects.
• Serving splits into retrieval (ANN over item vectors) and ranking (heavier per-candidate model with cross-features), and this pattern generalizes to any modern recommender.
• Depth delivers on cold items and long-tail catalogs; it does not fix a mis-framed problem or a wrong metric, and it costs more to run.

Next module: what to do the day a user or an item shows up with zero interactions — cold start, and why it is a design problem as much as a modeling one.