bleuscore

bleuscore is a fast BLEU score calculator written in rust. You can check try the web demo here for a quick experience.

Installation

The python package has been published to pypi, so we can install it directly:

• uv: uv add bleuscore
• pip: pip install bleuscore

Quick Start

The usage is exactly same with huggingface evaluate:

- import evaluate
+ import bleuscore

predictions = ["hello there general kenobi", "foo bar foobar"]
references = [
    ["hello there general kenobi", "hello there !"],
    ["foo bar foobar"]
]

- bleu = evaluate.load("bleu")
- results = bleu.compute(predictions=predictions, references=references)
+ results = bleuscore.compute(predictions=predictions, references=references)

print(results)
# {'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 
# 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}

Benchmark

TLDR: We got more than 10x speedup when the corpus size beyond 100K

We use the demo data shown in quick start to do this simple benchmark. You can check the benchmark/simple for the benchmark source code. Detailed per-PR results are tracked automatically via the Benchmark workflow.