An automatic, precision-oriented metric for evaluating machine translation output by comparing n-gram matches between a candidate translation and one or more reference translations; it aggregates n-gram precisions (typically via geometric mean) and applies a brevity penalty to penalize excessively short candidates.

Principle

Combine lexical n-gram precision across multiple orders into a single summary score while adjusting for length mismatch; higher n-gram overlap and closer length parity raise the score.

Demonstration

Illustrative scenario → Situation: a system produces a candidate translation for a corpus of source sentences and reference translations are available. Recognition: compute unigram, bigram, ... n-gram precisions between candidate and references and the candidate/reference length ratio. Action: aggregate precisions (geometric mean) and apply the brevity penalty. Consequence: the corpus-level BLEU score ranks systems by lexical overlap and length parity for that corpus, enabling comparative evaluation.

Misapplication

Using BLEU as a reliable sentence-level adequacy or fluency judge, or interpreting a low score with a single reference as definitive; the error is assuming BLEU's corpus-level precision summary directly reflects sentencewise semantic adequacy or covers paraphrase variation.

Consequence

When used for development or selection, BLEU steers optimization toward higher lexical overlap and length-matching with available references; when reported without context (number of references, corpus size), it can mislead about real-world translation quality.

Reversal

BLEU's aggregation and brevity penalty lose intended interpretability for very small datasets, single references, highly abstractive paraphrases, or language pairs where equivalent meaning is expressed with large lexical variation; under those conditions BLEU can systematically undervalue acceptable outputs.

Boundary

Clearly within: comparative corpus-level evaluation of MT systems with one or more reference translations. Boundary case: single-sentence evaluation with one reference (score becomes unreliable). Clearly outside: direct measures of semantic adequacy, fluency, or human acceptability without supplementary assessment.

Semantic Tension

Precision-oriented lexical overlap (BLEU) ↔ metrics or judgments that emphasize recall, semantic equivalence, or fluency; reliance on BLEU trades sensitivity to lexical matching for reduced sensitivity to paraphrase and meaning-preserving variation.

Synthesis

BLEU is a compact, corpus-level indicator of lexical overlap and length parity useful for system comparison under controlled reference conditions, but it is not a standalone measure of translation adequacy or fluency and must be complemented by other metrics or human evaluation.