Skip to main content

Module 4 — Content-based filtering and similarity

Collaborative filtering asks who liked what. Content-based filtering asks what is this thing about. The two families exist because they fail on opposite failure modes. Collaborative dies when a user or item has few interactions (modules 2 and 3 both show this). Content dies when descriptions are shallow or generic. This module writes the second family cleanly on our catalog and prepares its combination with the first, which is module 5.

The idea in one line​

Each item is turned into a vector — its content embedding — computed from what we know about the item: title, description, curriculum, tags, difficulty level, language. Each user is then represented as an aggregate of the vectors of items they liked. Scoring a candidate item for a user is a cosine similarity between the two vectors. That is the entire method.

Two things separate a good content-based recommender from a bad one: the quality of the embedding, and the way user profiles are built from interactions. The rest is bookkeeping.

Embeddings that carry meaning​

The vector representation is the whole game. We assume the reader has course 13 (NLP) or is comfortable with the vocabulary. Two families work in practice.

TF-IDF on the course description remains a surprisingly strong baseline. It captures topical vocabulary — "kubernetes", "transformer", "sarima" — and its cosine similarity is well understood. Its weakness is exact match: it does not know that "neural network" and "deep learning" are close.

Sentence embeddings from a modern encoder (Sentence-BERT and its multilingual variants, intfloat/multilingual-e5-large for our catalog) produce a fixed-size vector per item that captures topical and semantic content. On our platform, moving from TF-IDF to e5-base moved recall@10 on new items from 6 % to 14 %, at the cost of about 400 ms per new item at indexing time.

import numpy as np
import pandas as pd
from sentence_transformers import SentenceTransformer

items = pd.read_parquet("catalog/items.parquet")  # item_id, title, description, tags
# Concatenate the fields that carry actual meaning
texts = (
    items["title"] + ". "
    + items["description"].fillna("") + ". Tags: "
    + items["tags"].fillna("").str.replace("|", ", ")
).tolist()

encoder = SentenceTransformer("intfloat/multilingual-e5-base")
X = encoder.encode(
    ["passage: " + t for t in texts],
    normalize_embeddings=True,
    batch_size=64,
    show_progress_bar=True,
)          # shape: items x 768

items["vec"] = list(X)
items.to_parquet("catalog/items_with_vec.parquet")

Two implementation details worth noting. The passage: prefix is required for the E5 family and moves the score of a well-matched pair by about 5 %. The normalize_embeddings=True gives us unit-norm vectors, so cosine similarity becomes a plain dot product, which we will exploit next.

The user profile: not a mean​

The naive user profile is the average of the embeddings of the items the user liked. It works, it is easy, and it silently smooths away everything that makes the user interesting. A learner who took three courses on Kubernetes and two on Python ends up with a profile in the middle of the two topics, and both individual interests get diluted.

Two adjustments help substantially.

  • Weight positives by strength: a completion counts more than an enrollment, an enrollment more than a view. Multiply the item vector by the confidence weight before averaging.
  • Recency decay: an item watched two years ago should count less than one from last week. A simple exponential decay with a half-life of six months captures the drift.
from datetime import datetime
import numpy as np

def user_profile(events_u: pd.DataFrame, item_vec: dict) -> np.ndarray:
    now = pd.Timestamp("2026-09-01")
    half_life_days = 180.0
    dt = (now - events_u["ts"]).dt.days.clip(lower=0).astype(float)
    weight = events_u["conf"].to_numpy() * (0.5 ** (dt.to_numpy() / half_life_days))
    vecs = np.stack([item_vec[i] for i in events_u["item_id"]])
    v = (weight[:, None] * vecs).sum(axis=0)
    n = np.linalg.norm(v)
    return v / n if n > 0 else v

More sophisticated schemes exist (a small centroid per cluster of the user's interests, for instance, which gives noticeably better diversity), but the weighted, decayed mean is a strong baseline that ships in a day.

Scoring, ranking, and the exclusion set​

Once we have unit-norm user and item vectors, scoring a candidate is a dot product:

def top_k_content(user_vec: np.ndarray, item_matrix: np.ndarray, exclude: set[int], k: int = 10):
    scores = item_matrix @ user_vec
    scores[list(exclude)] = -np.inf
    return np.argsort(-scores)[:k]

At the scale of 500 items this runs in microseconds. At 100 000 items we would introduce FAISS or HNSW here for approximate nearest neighbors; the code becomes a two-liner around the same idea.

The exclude set is again critical: never recommend a course the user already consumed. On top of that, in a content system, we typically also exclude items too close to what the user just watched. A learner who finished a Docker course does not want the five other Docker courses next; they want the natural continuation. Module 5's hybrid approach makes this rule easier to enforce.

The filter bubble, in one paragraph​

Content-based systems reinforce whatever the user has already consumed. That is their defining behavior. A learner who took one Python course gets ten Python courses next; a learner who watched one long-form documentary gets ten more long-form documentaries. Over months, the recommender narrows the user's horizon rather than widening it. This is the filter bubble, and it is the reason "content only" recommenders age poorly on any platform whose value proposition includes discovery.

Three practical mitigations:

  • Diversity re-ranking (MMR): after the top-kk scoring pass, re-rank to trade a bit of relevance for a bit of dissimilarity between the recommended items. Module 8 gives the formula.
  • Explicit exploration: reserve one of the ten slots for a random or popular-but-unfamiliar item. It costs a few percent of CTR and it saves the medium-term diversity.
  • Hybridization: combine with collaborative filtering, which sees co-consumption patterns the content model is blind to. This is the topic of module 5.
The filter bubble is not a moral objection, it is a business problem

Users whose recommendations narrow over time complete fewer courses, get less variety of certificates, and churn faster. The filter bubble is a metric — call it intra-list diversity or catalog coverage — and a deployed recommender that scores well on relevance and badly on those two loses money on the 90-day horizon. Measure it from day one.

When content wins outright​

Content-based filtering is not a fallback for cold start; it is the right choice in several situations that come up often on a learning platform.

  • Fresh items: a new course has zero interactions on day one, and CF cannot score it. Content-based ranks it correctly from its description alone.
  • Explanations: "we recommend this course because it covers Kubernetes and observability, which you completed last month" is trivial to produce from vector overlap and painful to produce from latent factors.
  • Domain-guided recommendations: on a curated professional-training catalog, editorial rules ("this certification path is followed by this bridge course") are easier to inject via item metadata than via CF signals.

The lesson is that "collaborative vs content" is not a fight; both live in a real system, complementary. Which brings us naturally to hybridization.

Summary​

  • Content-based filtering scores items by the cosine similarity between an item embedding (from title, description, tags) and a user profile built from consumed items.
  • Modern sentence embeddings substantially outperform TF-IDF on new items; the naive mean-of-vectors profile benefits from confidence weighting and recency decay.
  • Content systems create a filter bubble: relevance rises, diversity and long-term retention fall; measure diversity from day one, do not discover the problem in production.
  • Content wins outright on new items, on explanations, and on editorial-driven catalogs; the hybrid of the next module keeps the wins while dodging the failures.

Next module: how to combine collaborative and content signals so that each covers the other's blind spot — hybrid systems, and how to choose the combination form.