# rougers
[](https://github.com/SyntaxSpirits/rougers/actions/workflows/ci.yml)
[](https://pypi.org/project/rougers/)
[](https://crates.io/crates/rougers)
[](LICENSE)
ROUGE metrics in Rust, with scores identical to Google's
[`rouge-score`](https://pypi.org/project/rouge-score/), about 30 times faster on one core
and over 100 times faster on all of them.
`rouge-score` is the reference implementation behind Hugging Face `evaluate`, most
summarization papers and many LLM evaluation pipelines. It is written in pure Python
and becomes the bottleneck once you score tens of thousands of outputs. `rougers`
reproduces it exactly, down to the last bit of every float, including the tokenizer,
NLTK's Porter stemmer and the summary-level `rougeLsum`.
## Python
```sh
pip install rougers
```
Change the import and keep the rest of your code:
```python
from rougers import rouge_scorer # was: from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL", "rougeLsum"], use_stemmer=True)
scores = scorer.score(
"The quick brown fox jumps over the lazy dog",
"The quick brown dog jumps on the log.",
)
scores["rouge1"]
# Score(precision=0.75, recall=0.6666666666666666, fmeasure=0.7058823529411765)
```
`score_multi`, custom tokenizers, `split_summaries`, `scoring.BootstrapAggregator` and
`tokenize.tokenize` work the same way as in `rouge-score`, and scorers can be pickled
for use in worker processes.
To score many pairs, use `score_batch`. It runs on all CPU cores and releases the GIL:
```python
results = scorer.score_batch(references, predictions) # list of dicts, one per pair
```
Set `RAYON_NUM_THREADS` to limit the number of threads.
## Rust
```toml
[dependencies]
rougers = "0.1"
```
```rust
use rougers::{RougeScorer, RougeType};
let scorer = RougeScorer::new(["rouge1", "rougeL"], true)?;
let scores = scorer.score(
"The quick brown fox jumps over the lazy dog",
"The quick brown dog jumps on the log.",
);
let rouge_l = scores.get(RougeType::L).unwrap();
println!("{:.4}", rouge_l.fmeasure);
```
`RougeScorer` is `Send + Sync`, so it can be shared across threads, for example with
Rayon.
## Performance
10,000 pairs of English text from the NLTK Gutenberg corpus, about 60 words each,
with `rouge1`, `rouge2`, `rougeL`, `rougeLsum` and stemming. Measured on an Apple M1 Pro
with Python 3.12:
| `rouge-score` 0.1.2 | 16.95 | 590 | 1× |
| `rougers` `score()` | 0.59 | 17,078 | 29× |
| `rougers` `score_batch()`, 8 threads | 0.12 | 83,957 | 142× |
The script asserts that all three produce the same scores. To reproduce:
```sh
cd bindings/python
python benchmarks/compare.py --pairs 10000
```
## Compatibility
The test suite compares `rougers` with `rouge-score` and fails on any difference in
any float:
- the stemmer against NLTK on all 235,976 words of a system dictionary (NLTK 3.8–3.10
stem them identically);
- tokenization against Python's `str.lower` for every Unicode code point;
- thousands of random texts with repeated words, punctuation, digits, newlines and
non-Latin scripts, plus property-based tests on arbitrary Unicode;
- `score_multi`, custom tokenizers, `split_summaries` and bootstrap aggregation with
a fixed NumPy seed.
Known differences:
- An invalid ROUGE type raises `ValueError` when the scorer is created, not on the
first call to `score`. Type names must be exactly `rouge1`…`rouge9`, `rougeL` or
`rougeLsum`.
- Texts must be `str`; `rouge-score` also accepts UTF-8 `bytes`.
- `BootstrapAggregator` needs NumPy (`pip install 'rougers[aggregate]'`) and
`split_summaries=True` needs NLTK (`pip install 'rougers[sentences]'`). Without
them, `rougers` has no Python dependencies.
## Development
```sh
cargo test -p rougers
cd bindings/python
uv sync --group dev
uv run pytest
```
## License
Apache-2.0, the same licence as `rouge-score` and NLTK, whose scoring, tokenization and
stemming logic this project ports. See [NOTICE](NOTICE).