BLEU Score

BLEU scores machine-generated text by measuring how many word sequences it shares with one or more human reference translations. It runs from 0 to 100, and higher is better.

It became the standard for translation because it is automatic, cheap and repeatable. It is also blunt: it rewards matching the reference wording rather than conveying the meaning.

Think of it like this. Think of marking an essay by counting phrases shared with the model answer. Fast and consistent, and it scores a correct answer in different words the same as a wrong one.

How it works

• Compare n-grams, overlapping sequences of one to four words, between output and reference.
• Measure precision: how much of the generated text appears in the reference.
• Apply a brevity penalty, otherwise very short output would score well by saying almost nothing.
• Combine into 0 to 100. Above 40 is generally considered good for translation.

Common mistakes

• "High BLEU means good translation." It means wording similar to the reference. A perfect translation using different words scores badly.
• "Compare BLEU across papers." Different tokenization and different reference sets change it. Only sacreBLEU-style standardised settings are comparable.
• "Use it for any generation task." It was built for translation, where a reference is meaningful. For open-ended generation there is no single right answer to match.
• "It is still the state of the art." Learned metrics such as BERTScore and COMET correlate far better with human judgement.