I Built a Recommendation Engine in 30 Minutes (And So Can You)

A system that recommends things by comparing users with similar interest are everywhere in social media.

“If similar users liked this, you may like it too”

this is simple collaborative filtering

Most of the current CF used something called memory-based CF, system recommends by directly comparing users or items from stored ratings.

Similarity & Distance

Before recommendation technique familiarizing this similarly and distance manifestly plays a paramount role.

Pearson correlation

How strongly 2 things move together?

Deriving Pearson correlation might give you broad understanding, recommend you that, but don't feel bewilderment when you can't.

Jaccard similarity

Measures how many things two groups are in common

Spearman rank correlation

Measures whether 2 things follow a similar ranking a pattern.

So, my ranking

1. AI
2. Marvel
3. DC

it might be totally unrelated as well much similar, it's arduous to understand.

Concept is if you want to find our users similar taste use Pearson correlation, but if you find the common item what they like, use Jaccard similarity , most of the users has a particular ranking to find out use Spearman task correlation.

Vector normalization

Dot product

Mean centering

Now the data centered around 0.

patterns became easier to detect, if with mean centering.

large offsets confuse ML algorithm if without mean centering, its efficacious.

let's balance around a common centre first.

Manhattan distance

Euclidean distance

Principles of User-Based and Item-Based Collaborative Filtering

User-Based Collaborative Filtering

• Core principle: Find users similar to the target user (neighbors), then recommend items those similar users liked but the target hasn’t seen.
• Steps:

Build a user–item rating matrix.

Compute similarity between users (e.g., cosine, Pearson correlation).

Select top k most similar users (neighborhood).

Aggregate their ratings (weighted average) to predict rating for a target item.

Recommend highest-predicted items not yet rated by target user.

Item-Based Collaborative Filtering

• Core principle: Find items similar to those the target user has liked, then recommend those similar items.

Steps:

Build a user–item rating matrix.

Compute similarity between items (using co-ratings from multiple users).

For a target user and item, find items the user has rated that are similar to the target item.

Predict rating as weighted average of user’s ratings on similar items.

Recommend items with highest predicted ratings.

User collaborative filtering

Item-item collaborative filtering

Limitations of Traditional Recommendation Systems

When data became large, sparse or constantly changing it will be arduous.

Let's say a user without no watch history or no ratings?

most users when rate very few items?

Comparing millions of users or even item became computationally expensive.

let's say a page ranked individual, popular item dominate.

Spotify recommends gym music at midnight while users want sleep music.

Fake reviews by influencer who are high in page rank boosts a bad product.

Over specialization do exist, to exemplify if you watch one superhero movie let's say home lander, entire homepage became superheroes

Scalability

• User‑based CF: Similarity computation among all users → O(N2M) O(N2M) for dense matrix (N users, M items). As N grows to millions, real-time prediction becomes infeasible.
• Item‑based CF: O(M2N) O(M2N) — better if M << N, but e-commerce sites have millions of items.
• Nearest neighbor search without indexing (e.g., LSH) is slow; precomputing similarity requires O(N2) O(N2) storage.

Case Studies Demonstrating Traditional Recommendation Systems

Gray sheep problem does exist.

user_preference = [
"Extreme horror",
"Documentary"
]
print("Difficult to find similiar users")

as well as shilling attack, fake users manipulate ratings to trick recommendation system.

TF-IDF

learning TF-IDF might be efficacious, term frequency inverse document frequency.

Inverse popularity weighting

Overly popular items like lets many influencers unwanted promotion or other products can reduce influence using current Modern RS system, that is non arduous now.

Viral video gets down weighted in personalization.

Propensity

Sometime if we know the probability or tendency of an action happening, to exemplify Propensity it plays a paramount role, at that point relegate will be optional.

When we figure out likely item that a user buy, i mean high propensity in the case of e commerce or ad click prediction, it's much efficacious at this point. ( propensus = leaning forward)

Bandit algorithm

Bandit algorithm also really helps, a system that learns which option is best by trying and adjust over time.

This system learns which option is best trying and adjusting over time, try different post learning engagement is non arduous, like it's a part of conditioning over time like Bandit

like in arm a, b, c each gives a different reward.

epsilon greedy strategy is also a best, mostly best option, sometime explore new ones, to exemplify multiple treatments gradually prefer best one.

A self-model goes in an exam time our maximum exploitation but necessary not an exploration.

A system to work really well it should go from gaining provenance data in exploration perspective same as in unsupervised learning to an exploitation in real world play for maximum efficiency. at the bandit play, there is no time to exploration, only exploitation. We can do imputation in our self-model during exploration stage, each time filling necessary part as a part of conditioning for maximum output.

ANNOY and HNSW

approximate nearest neighboring oh yeah and Hierarchical Navigable Small world.

first concept is building many random trees to quickly narrow search space and in 2nd you finding people through friends of friends efficiently.

Both avoid checking of all vectors O(n)

HNSW > ANNOY

Weighted average prediction formula, like user-based CF is efficacious to predict how a user will rate an item, take ratings from a similar user, multiply each rating by how similar they are finally normalizing by total similarity.