An Intro to Ranking Metrics : How Good Is Your Recommender System?

ML Quickies #61

When Netflix recommends a movie, or Spotify suggests a song, how do we know if the system is actually doing its job? When it comes to recommendations, simply being accurate is not enough, the order matters too. Showing the right movie in position 10 is very different from showing it in position 1.

In this post, we will go through the most important ranking metrics used to evaluate recommender systems, using a movie recommendation system as a running example. All of these metrics share a common parameter: K, the number of items shown to the user (e.g., a top-10 recommendations list).

Imagine we’ve built a movie recommender. For a given user, we know (from their watch history) which movies they actually enjoyed, call these the relevant items. Our model produces a ranked list of recommendations. We want to know: how well does our ranked list match what the user actually likes?

Let’s say the user likes 5 movies, and our model returns the top 5 recommendations, in order:

Recommended:   [A, B, C, D, E]
User liked:    [A, C, X, Y, Z]

In this example, A and C are hits; and movies B, D, E are misses. Now let’s see what metrics we can use to quantify this.

1. Precision@K : Are The Recommendations Relevant?

Precision@K : of the K items recommended, what fraction were actually relevant?

In our example with K=5, the model recommended [A, B, C, D, E] and A and C are relevant. So:

Precision@5 = 2/5 = 0.4

Strengths: Simple and intuitive. If your goal is precision (“every recommendation should count”), this is the right metric.

Weaknesses: It doesn’t care about order. Recommending [A, C, B, D, E] gives the same Precision@5 as [B, D, E, A, C], even though the first ordering is much better as it lines up the good stuff early.

2. Recall@K : Am I Finding What the User Likes?

Recall@K : of all the items the user actually likes, what fraction did come up in the top-K?

In our example, the user likes 5 movies and we found 2 of them (A and C) in our top-5:

Recall@5 = 2/5 = 0.4

Strengths: It tells you about coverage ie how well you’re retrieving the user’s full taste profile.

Weaknesses: Like Precision, it also ignores order. Also, it’s easy to inflate recall by simply recommending everything but that of course, destroys precision.

The precision-recall tradeoff: A system that recommends only 1 very safe item might have high precision but terrible recall. A system that recommends 100 items will catch almost everything the user likes (high recall) but will also recommend a lot of junk (low precision). Therefore, we can think of K as the knob you turn.

3. Hit Rate@K a.k.a. Recall@K for single-item relevance

Hit Rate@K: did at least one relevant item appear in the top-K list?

This is typically averaged across all users. So if 80 out of 100 users had at least one relevant item in their top-K recommendation list, the system’s Hit Rate@K is 0.80.

When to use it: Hit Rate is ideal for scenarios where a single good recommendation is a “win”, think push notifications, email digests, or homepages. It’s also a very forgiving metric: even a mediocre system will usually score reasonably well on Hit Rate.

4. Normalized Discounted Cumulative Gain

NDCG@K: This is the gold standard metric when position matters. The core idea : relevant items that appear earlier in the list are worth more than those buried at the bottom.

We will break it down step by step.

Step 1:Discounted Cumulative Gain (DCG)

Each item’s contribution is discounted by its position using a logarithm:

Where rel_i is 1 if the item at position i is relevant, 0 otherwise.

For our example we had : [A✅, B❌, C✅, D❌, E❌]:

Step 2: Ideal DCG (IDCG)

What’s the best possible DCG we could get? That’s the DCG of the perfect ranking, all relevant items first. With 2 relevant items in top-5, the ideal ordering is [A✅, C✅, , , _]:

Step 3: Normalize

If movie C had appeared at position 5 instead of position 3, DCG would drop to 1.0 + 0.43 = 1.43, and NDCG would be ~0.88. That’s the ranking penalty in action.

Strengths: NDCG@K captures both relevance and position. Its widely used in industry and research. It can also support graded relevance (e.g., “loved it” = 3, “liked it” = 2, “watched it” = 1) by replacing binary rel_i with a relevance score.

Real systems rarely have crisp “relevant / not relevant” signals. A movie a user watched for 5 minutes is not the same as one they rewatched three times and rated 5 stars. That’s where graded relevance comes in.

Metrics like Precision@K and Recall@K require you to pick a threshold (e.g., “4 or 5 stars counts as relevant”), which always involves a subjective judgment call.

Weaknesses: Harder to interpret intuitively. Requires knowing the ground-truth relevance, which can be expensive to collect.

5. Mean Reciprocal Rank (MRR@K)

MRR@K: where does the first relevant item appear?

It’s particularly useful when only the first hit matters to the user. For example: in a search engine or autocomplete lists.

For a single user, if the first relevant item (A) appeared at rank 1:

If A had appeared at rank 3:

This is averaged across all users to get MRR. If a user has no relevant items in the top-K, their RR is 0.

Which Metric Should You Use?

There’s no single right answer as it depends on your product goals:

• Use Precision@K when every slot on screen is precious and you want to minimize irrelevant recommendations (e.g., hero section of the homepage with only 3 items).
• Use Recall@K when you want to surface a user’s full taste profile (eg: important in catalog exploration or when users are likely to scroll far).
• Use Hit Rate@K for top-of-funnel evaluation. It’s especially meaningful for push notifications or “because you watched X” recommendations where a single good suggestion is a win.
• Use NDCG@K as your go-to benchmark metric during model development and A/B testing. It captures the full quality of your ranked list , both the quality of recommendations and and in what order they appeared.
• Use MRR@K when the use case is closer to search than discovery ie where the user has a specific need and will stop at the first relevant result.

How to Choose K?

The choice of K is not cosmetic as it can fundamentally alter what each metric measures:

• A very small K (e.g., K=1) is unforgiving. Even a great model will have low recall simply because it can’t surface many items.
• A large K (e.g., K=50) inflates recall and hit rate, but users rarely scroll that far. Your metric no longer reflects real behavior.

A useful rule of thumb: set K to match the actual UI interface. If your movie carousel shows 10 items, evaluate at K=10. If your notification shows 1 item, MRR@1 or Hit Rate@1 would be the honest metric.

Ranking metrics are the lens through which you see whether your recommender is actually helping users, or if it is just generating confident-sounding noise.

No single metric tells the whole story. In practice, teams track NDCG as the headline number, Precision and Recall for diagnostics, and Hit Rate for a sanity check on coverage. Watching all of them together, and watching how they move relative to each other when you ship a new model is where the real intuition comes from.

Until next time:)