ROUGE Score

ROUGE scores generated summaries by measuring how much of a human reference summary appears in the output. It emphasises recall, meaning coverage of the reference.

That emphasis is the difference from BLEU. In summarisation, missing an important point matters more than adding a few extra words, so the metric rewards covering the reference.

Think of it like this. Think of marking a summary against a checklist of points that should be mentioned. You are scoring what was covered, not penalising every extra sentence.

Types

• ROUGE-N. Overlap of n-grams. ROUGE-1 is single words, ROUGE-2 is word pairs.
• ROUGE-L. Longest common subsequence, which rewards matching order without demanding contiguity.
• ROUGE-W. Weighted version favouring longer consecutive matches.
• Report several. ROUGE-1, ROUGE-2 and ROUGE-L together is the convention, since each catches something different.

Common mistakes

• "High ROUGE means a good summary." It means word overlap with the reference. A summary can score well while being incoherent, and a well-written one using different words scores badly.
• "It measures factual accuracy." It does not. A summary can hallucinate and still score well by reusing the reference's vocabulary.
• "One ROUGE number is enough." Report ROUGE-1, ROUGE-2 and ROUGE-L. A high ROUGE-1 with low ROUGE-2 usually means the right words in the wrong order.
• "It works for any generation task." It needs a reference. Open-ended generation has no single right answer to compare against.