Collaborative Filtering: Wisdom of the Crowd, Personalised for You

Imagine walking into a bookstore and asking the staff: “I loved Dune — what should I read next?” The best recommendations don’t come from analyzing the book’s plot keywords. They come from knowing what thousands of other Dune fans went on to enjoy. That intuition — people who agree with you are your best guides — is the beating heart of collaborative filtering.

Unlike content-based filtering, which studies the items themselves, collaborative filtering studies patterns of human behaviour. It doesn’t need to know a single thing about what a movie is about, as long as it knows who rated it — and how.

The big idea

The core assumption is beautifully simple: if User A and User B have rated many items similarly in the past, they’ll probably agree on new items too.

Two flavours of collaborative filtering

Item-based CF tends to be more scalable and stable in practice — items change far less often than user preferences. Amazon pioneered item-based CF at scale.

The rating matrix: where it all starts

The raw material is a matrix R where rows are users, columns are items, and each cell is a rating (or blank if the user hasn’t rated that item). The job of CF is to fill in the blanks.

The mathematics

1. Pearson correlation: measuring user similarity

We need to quantify how alike two users are. Pearson correlation is the standard choice — it compares relative preferences rather than raw scores, handling the fact that some people rate everything a 5 while others never go above a 3.

sim(u, v) = Σ(r_ui − r̄_u)(r_vi − r̄_v) / √[Σ(r_ui − r̄_u)² · Σ(r_vi − r̄_v)²]

Sum runs over items i rated by both users u and v

r_ui = rating user u gave to item i

r̄_u = average rating of user u (across all rated items)

sim(u, v) = −1 (opposite taste) → 0 (no correlation) → +1 (identical taste)

2. User-based prediction formula

Once we have similarity scores, we predict a user’s rating for an unseen item by taking a weighted average of similar users’ ratings for that item:

r̂_ui = r̄_u + Σ sim(u,v) · (r_vi − r̄_v) / Σ |sim(u,v)|

Sum over the top-K most similar users v who rated item i

r̂_ui = predicted rating of user u for item i

r̄_u = mean rating of user u (bias correction)

sim(u, v) = similarity between users u and v

The mean-centering step (r_vi − r̄_v) is crucial — it corrects for users who systematically rate high or low, so we compare relative preferences rather than raw numbers.

3. Item-based cosine similarity

For item-based CF, we treat each item as a vector of user ratings and measure cosine similarity between item vectors:

sim(i, j) = (R_i · R_j) / (‖R_i‖ · ‖R_j‖)

R_i = vector of all users’ ratings for item i

Matrix factorization: the modern approach

Memory-based CF (user/item similarity) works well for small datasets. But for millions of users and items, we need something smarter. Enter matrix factorization — the technique behind Netflix Prize winners and modern recommendation engines.

The idea: decompose the sparse rating matrix R into two smaller dense matrices — a user matrix P and an item matrix Q — such that their product approximates R:

R ≈ P × Qᵀ

P = Users × Latent factors matrix (each user mapped to K hidden dimensions)

Q = Items × Latent factors matrix (each item mapped to K hidden dimensions)

K = number of latent factors (typically 20–200)

4. The objective function (SGD / ALS)

We learn P and Q by minimizing the prediction error on known ratings, with L2 regularization to prevent overfitting:

min Σ (r_ui − pᵤᵀqᵢ)² + λ(‖pᵤ‖² + ‖qᵢ‖²)

Sum over all observed (u, i) pairs

pᵤ = latent factor vector for user u (K-dimensional)

qᵢ = latent factor vector for item i (K-dimensional)

λ = regularization strength (prevents overfitting)

This is solved iteratively with Stochastic Gradient Descent (SGD) or Alternating Least Squares (ALS). The latent factors learn abstract concepts — things like “taste for cerebral sci-fi” or “preference for action-heavy plots” — without ever being told what those concepts are.

Intuition check — if user Alice has a high score on latent factor 3 (secretly learned as “loves cerebral sci-fi”), and Interstellar also has a high score on factor 3, their dot product is large → high predicted rating. The model discovers these hidden taste dimensions automatically from the rating patterns alone.

Step-by-step: the full pipeline

1. Collect interaction data

Gather explicit ratings (stars, thumbs) or implicit signals (clicks, watch time, purchases).

2. Build the rating matrix

Construct the Users × Items matrix R. Most cells will be empty — typical sparsity is 99%+.

3. Compute similarity (or factorize)

For memory-based CF: compute pairwise Pearson/cosine similarity. For model-based CF: run SGD/ALS to find P and Q.

4. Generate predictions

For each (user, unseen item) pair, compute a predicted score using the weighted neighbour formula or the dot product pᵤᵀqᵢ.

5. Rank and serve top-N

Sort predicted scores. Surface the highest-ranked items the user hasn’t interacted with yet.

Pros and cons

Where is it used today?

Collaborative filtering is arguably the most impactful algorithm in consumer technology — responsible for a third of Netflix viewing and 35% of Amazon purchases. When combined with content-based methods in a hybrid system, it forms the backbone of virtually every recommendation engine you encounter daily.