The Metric That Grades AI Translations -And Why It’s Both Brilliant and Broken

How a 2002 paper from IBM researchers became one of the most cited and controversial tools in AI.

You’ve built a machine translation system. The output looks reasonable. But is it actually good?

You could hire translators to evaluate it. That takes weeks and costs a fortune. Or you press a button and get a number in seconds.

That number is BLEU. And for 20+ years, it’s been the default way to measure machine translation quality.

The GAME ?

BLEU counts how many n-grams (word sequences) the machine output shares with a human reference translation.

Reference: “The cat sat on the mat.” Candidate A: “The cat is sitting on the mat.” ✅ Candidate B: “The cat the cat the cat.” ❌

Candidate B tries to cheat by repeating words that appear in the reference. BLEU blocks this with clipping. You only get credit for a word as many times as it appears in the reference. No gaming the system.

How It Works ?

Step 1: Modified Precision

For each n-gram order n, compute the clipped precision:

pₙ = Σ Count_clip(n-gram) / Σ Count(n-gram)

In words: out of all n-grams the machine produced, what fraction also appeared in the reference (capped)? Do this for 1-grams through 4-grams.

Step 2: Combine with a geometric mean

BLEU blends all four precision scores using a log-average:

log(BLEU) = BP + Σ wₙ · log(pₙ)    where wₙ = 1/4

Longer n-grams (3-grams, 4-grams) are harder to match and capture fluency and grammar — so getting them right really matters.

Step 3: Brevity Penalty (BP)

A system that outputs just one correct word would score 100% precision. BLEU penalizes short outputs:

BP = 1                  if c > r
BP = exp(1 - r/c)       if c ≤ r

where c is candidate length and r is reference length. Output too short? Your score gets deflated exponentially.

Step 4: Final score between 0 and 1

BLEU = BP · exp(Σ wₙ · log(pₙ))

A score of 1.0 means a perfect match. In practice, scores above 0.4 are considered strong for machine translation.

One key thing: BLEU is designed to work across a corpus, not on single sentences. Averaged over thousands of examples, it correlates reasonably well with human judgment.

Where It Breaks ?

• “Large” ≠ “big”: synonyms get no credit
• Multiple valid translations: BLEU penalizes correct alternatives it didn’t see
• Fluency failures: grammatically wrong but word-overlapping outputs can score well
• Meaning : two sentences can share n-grams while meaning totally different things

Why It Still Dominates ?

Better metrics exist, METEOR handles synonyms, BERTScore uses neural embeddings to measure meaning, COMET trains directly on human judgments.

Yet BLEU persists because it’s fast, free, and universal. Every team can compute it the same way. Every reader knows what the number means. When Google trains a new translation model through hundreds of iterations, BLEU runs automatically on every single one a smoke alarm, not a final verdict.

It’s the metric everybody criticizes and nobody stops using.

Conclusion

BLEU is a 2002 shortcut to a real, enduring problem. It’s imperfect in predictable ways, which makes it useful if you know its limits.

Based on “BLEU: A Method for Automatic Evaluation of Machine Translation” — Papineni et al., ACL 2002.

Tags: NLP, Machine Learning, AI, Machine Translation