BLEU vs ROUGE vs METEOR vs BERTScore: Bridging the Gap Between Lexical and Semantic Evaluation in NLP

Introduction

Let’s start with a simple example.

Reference sentence:

“The cat is sitting on the mat”

Model prediction:

“A cat is resting on a rug”

If you read both sentences, they clearly mean the same thing. The subject is the same, the action is similar, and the overall idea hasn’t changed. Any human reading this would say the prediction is correct.

But when we evaluate this using common NLP metrics like BLEU or ROUGE, the score turns out to be low.

That feels wrong. Why does this happen?

The reason is simple: most traditional evaluation metrics don’t actually understand meaning. They only look at exact word matches.

So in this case:

• “sitting” and “resting” are treated as different
• “mat” and “rug” are treated as unrelated
• even small changes like “the” vs “a” affect the score

Because of this, even a perfectly valid paraphrase can get penalized. This shows a fundamental problem in how we evaluate NLP models.

The Real Issue

There is a clear mismatch between how humans and metrics judge language:

• Humans focus on meaning
• Metrics focus on word overlap

This mismatch is what we call the lexical–semantic gap.

In earlier NLP systems, this wasn’t a big issue because models mostly generated text close to the reference. Exact matching worked reasonably well.

But modern models are different. They can:

• paraphrase naturally
• use synonyms
• restructure sentences
• generate more human-like responses

So What’s The Problem?

This leads us to an important question:

Why do our evaluation metrics fail to capture meaning?

To answer this, we need to look deeper into how these metrics actually work and what they are really measuring.

In this article, we’ll explore four widely used evaluation metrics — BLEU, ROUGE, METEOR, and BERTScore — and understand how each of them tries (or fails) to bridge the gap between word-level matching and true semantic understanding.

The Core Problem : Evaluation in NLP

When we build NLP models, one thing sounds simple at first: “Just compare the model output with the correct answer.”

But in reality, it’s not that straightforward. Unlike tasks like math or coding, language doesn’t have just one correct answer. For a single sentence, there can be many valid ways to say the same thing.

For example:

• “The meeting was postponed due to rain”
• “The meeting got delayed because of the rain”
• “Rain caused the meeting to be rescheduled”

All of these are correct. All of them carry the same meaning. But if your model generates one version and your reference has another, most metrics will still penalize it. This is the core issue.

In most of the NLP tasks, there is no single ground truth. Instead, there are many possible correct outputs. But our evaluation systems usually rely on just one reference sentence.

So the model is judged not on whether it is correct, but on whether it is similar to that one reference.

Limitations

Most popular metrics (like BLEU, ROUGE, etc.) follow a reference-based evaluation approach:

Compare the model output with a fixed reference and calculate similarity.

This creates a big limitation:

• If your output is correct but phrased differently → low score
• If your output copies the reference words → high score

So the metric ends up rewarding surface similarity, not actual understanding. To understand this better, we need to look at two key ideas:

1. Lexical Matching

This is what traditional metrics focus on and works like:

• It checks exact word overlap
• Order and frequency of words matter
• Synonyms are treated as different

Example:

• “car” vs “automobile” → considered different
• “big house” vs “large house” → partially matched

2. Semantic Matching

This is closer to how humans evaluate language.

• It focuses on meaning, not exact words
• Synonyms and paraphrases are understood
• Context matters

Example:

• “car” vs “automobile” → same meaning
• “sitting” vs “resting” → similar idea

Right now, most evaluation methods are still biased toward lexical matching, while modern NLP models are moving toward semantic understanding. That’s the mismatch.

And unless we fix how we evaluate, we might:

• underestimate good models
• overestimate shallow ones
• and miss real progress in NLP

Taxonomy of Evaluation Metrics

Now that we understand the core problem, let’s organize the major evaluation metrics into a simple taxonomy.

Not all metrics are built the same. Some just count word overlap, while others try to understand meaning. We can divide them into three broad categories:

1. Lexical Metrics (Surface-Level Matching)

These are the most traditional metrics. They focus purely on word overlap between the generated text and the reference.

BLEU (Bilingual Evaluation Understudy)

BLEU is mainly used in machine translation.

Core idea: Count how many n-grams (word sequences) match between prediction and reference.

Formula:

BLEU Formula

from nltk.translate.bleu_score import sentence_bleu

reference = [["the", "cat", "is", "sitting", "on", "the", "mat"]]
candidate = ["a", "cat", "is", "resting", "on", "a", "rug"]

score = sentence_bleu(reference, candidate)
print(score)

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

ROUGE is mostly used in text summarization.

Core idea: Measure how much of the reference is covered by the prediction.

Formula:

ROUGE Formula

from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)

reference = "the cat is sitting on the mat"
candidate = "a cat is resting on a rug"

scores = scorer.score(reference, candidate)
print(scores)

2. Semi-Semantic Metrics

These try to fix some limitations of lexical metrics by adding linguistic knowledge.

METEOR (Metric for Evaluation of Translation with Explicit ORdering)

Core idea: Align words using exact match, stem match or synonym match.

Formula:

METEOR Formula

from nltk.translate.meteor_score import meteor_score

reference = "the cat is sitting on the mat"
candidate = "a cat is resting on a rug"

score = meteor_score([reference], candidate)
print(score)

3. Fully Semantic Metrics

These are more modern approaches that use deep learning embeddings to capture meaning.

BERTScore

Built on top of transformer models like BERT.

Core idea: Instead of comparing words directly, compare their vector representations (embeddings).

Each word is converted into a high-dimensional vector that captures its meaning in context.

Formula:

BERTScore Formula

Then F1 is computed from precision and recall.

from bert_score import score

cands = ["a cat is resting on a rug"]
refs = ["the cat is sitting on the mat"]

P, R, F1 = score(cands, refs, lang="en")
print(F1)

Deep Dive Into Metrics

Now let’s go one level deeper and actually see how each metric works, not just in theory, but in practice.

We’ll use a simple example across all metrics so you can clearly see the difference in behavior:

• *Reference*: “the cat is on the mat”
• *Prediction*: “the cat sat on the mat”

4.1 BLEU (Precision-Based)

BLEU checks how many n-grams (word sequences) in the prediction match the reference.

It also applies a brevity penalty if the sentence is too short.

import nltk
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction

nltk.download('punkt')

reference = [["the", "cat", "is", "on", "the", "mat"]]
candidate = ["the", "cat", "sat", "on", "the", "mat"]

smooth = SmoothingFunction().method1

score = sentence_bleu(reference, candidate, smoothing_function=smooth)
print("BLEU Score:", score)

4.2 ROUGE (Recall-Based)

ROUGE measures how much of the reference is covered by the prediction.

!pip install rouge_score
from rouge_score import rouge_scorer

reference = "the cat is on the mat"
candidate = "the cat sat on the mat"

scorer = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)

scores = scorer.score(reference, candidate)

print("ROUGE-1:", scores['rouge1'])
print("ROUGE-L:", scores['rougeL'])

4.3 METEOR (Alignment-Based)

METEOR improves things by using:

• Stemming (run vs running)
• Synonyms (via WordNet)
• Better alignment

import nltk
from nltk.translate.meteor_score import meteor_score

nltk.download('wordnet')
nltk.download('omw-1.4')

reference = ["the", "cat", "is", "on", "the", "mat"]
candidate = ["the", "cat", "sat", "on", "the", "mat"]

score = meteor_score([reference], candidate)

print("METEOR Score:", score)

4.4 BERTScore (Semantic Evaluation)

BERTScore uses transformer embeddings.

Instead of comparing words:

• It compares meaning in vector space
• Uses cosine similarity

!pip install bert-score

from bert_score import score

cands = ["the cat sat on the mat"]
refs = ["the cat is on the mat"]

P, R, F1 = score(cands, refs, lang="en", verbose=True)

print("Precision:", P.mean().item())
print("Recall:", R.mean().item())
print("F1 Score:", F1.mean().item())

Quick Comparision

Now that we’ve understood each metric individually, let’s put them side by side. This is where things become very clear, each metric is built with a different goal in mind, and that directly affects how it behaves.

Comparison between multiple metrics

Cross-Task Evaluation

When we move from theory to real NLP tasks, the limitations of these metrics become very clear. The same metric that works fine in one task can fail badly in another.

Take machine translation. BLEU has been the standard metric here for years. It works okay when the generated sentence is very close to the reference. But modern models don’t translate word-by-word anymore — they rephrase naturally. So even if the meaning is perfectly correct, BLEU can give a low score just because the wording is different. For example, “completed the task” vs “finished the job” means the same thing, but BLEU will still penalize it. On the other hand, BERTScore handles this much better because it looks at meaning, so it recognizes that both sentences are semantically similar.

Now look at text summarization. ROUGE is the most commonly used metric here because it checks how much of the reference content is covered. This works well for older extractive methods where summaries are mostly copied from the original text. But with modern abstractive models, summaries are rewritten in a more natural way. That’s where ROUGE starts failing. A summary can be perfectly valid but still get a lower score just because it uses different words or structure.

So across both tasks, one thing becomes very clear:

Same meaning ≠ same words

And this is exactly where traditional metrics struggle. They assume that good output should look similar to the reference, while modern NLP models focus on generating text that means the same thing, not necessarily looks the same.

The Lexical vs Semantic Gap

At the heart of all this lies one core problem — the mismatch between how metrics evaluate language and how humans understand it.

Let’s go back to our example:

• Reference: “The cat is sitting on the mat”
• Prediction: “A cat is resting on a rug”

As humans, this feels completely correct. The meaning is the same. Nothing important has changed. But most traditional metrics will still give this a low score. Why? Because they are looking at the sentence in a very different way.

Lexical metrics like BLEU and ROUGE work at the surface level. They assume that if two sentences are similar, they should share the same words.

So they check:

• word overlap
• exact matches
• n-grams

And because of that:

• “sitting” ≠ “resting”
• “mat” ≠ “rug”

Semantic metrics, on the other hand, try to evaluate meaning. They don’t just look at words — they look at context.

So they understand that:

• “sitting” and “resting” are related
• “mat” and “rug” are similar in context

This is why methods like BERTScore give a much higher score for the same example.

Future Directions

If you look at the trend, evaluation in NLP is clearly moving in one direction — away from fixed rules and toward learned, meaning-aware systems.

Earlier metrics like BLEU and ROUGE were manually designed. They follow fixed formulas and assumptions. But now, newer approaches are trying to learn what good text actually looks like.

One important direction is learned evaluation metrics like COMET and BLEURT. These models are trained on human judgments, which means instead of just counting overlaps, they learn patterns of what humans consider “good” or “bad” output. In many cases, they correlate much better with human evaluation compared to traditional metrics.

Another interesting shift is LLM-based evaluation. Instead of writing formulas, we can now use large language models (like GPT-style models) to evaluate outputs. For example, you can ask a model:

“How similar are these two sentences in meaning?”

And it can give a score or explanation. This brings evaluation much closer to how humans think — but it also introduces challenges like bias, consistency, and cost.

Then comes human-in-the-loop evaluation, which is still the most reliable approach. No matter how advanced metrics become, human judgment remains the gold standard. The future is likely a hybrid system where:

• models evaluate quickly
• humans verify critical cases

So What To Actually Use?

If you’re building or evaluating an NLP system, don’t overcomplicate things. Use the right metric for the right job:

• BLEU: use it as a quick baseline, especially for machine translation
• ROUGE: use it for summarization tasks where content coverage matters
• METEOR: use it when you want a slightly better version of BLEU (handles synonyms, alignment)
• BERTScore: use it when you care about actual meaning and semantic similarity

Conclusion

Over time, NLP evaluation has gone through a clear shift.

We started with metrics that focused on word matching — counting overlaps, checking n-grams, and comparing exact phrases. That worked when models were simple and outputs were predictable.

But today, models generate language in a much more flexible and human-like way. They paraphrase, restructure, and express the same idea using different words. And this is where the old evaluation methods start to fall short.

The transition we’re seeing is important:

From word matching to meaning understanding

Metrics like BLEU and ROUGE still have their place, but they can’t fully capture what modern NLP systems are capable of. Newer approaches like BERTScore, COMET, and LLM-based evaluation are pushing us closer to evaluating language the way humans do.

But we’re not fully there yet. Evaluation in NLP is still evolving, and probably will continue to evolve as models become more advanced.

Implementation Notebook

If you want to explore the full implementation and reproduce the results: Full notebook available here

References Used

[1] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), 2002, pp. 311–318.

[2] C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” in Proceedings of the ACL Workshop on Text Summarization Branches Out, 2004, pp. 74–81.

[3] S. Banerjee and A. Lavie, “METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005, pp. 65–72.

[4] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating Text Generation with BERT,” in Proceedings of the International Conference on Learning Representations (ICLR), 2020.