pictor-eval 0.1.0

Model evaluation harness for Pictor — perplexity, MMLU, benchmarks
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

# pictor-eval

[![Version](https://img.shields.io/badge/version-0.2.2-blue.svg)](https://crates.io/crates/pictor-eval)
[![Status](https://img.shields.io/badge/status-stable-brightgreen.svg)]()
[![Tests](https://img.shields.io/badge/tests-513%20passing-brightgreen.svg)]()

Model evaluation harness for Pictor — ROUGE, perplexity, accuracy, throughput.

Provides perplexity measurement, MMLU-style multiple-choice accuracy,
ROUGE-N/L/S scoring, exact-match scoring, throughput benchmarking, JSONL
dataset loading, and JSON/Markdown report generation.

Part of the [Pictor](https://github.com/vbasky/pictor) project.

## Status

**Stable** (v0.2.2) — 513 tests passing.

## Features

- `PerplexityEvaluator` — from log-probs or logits; bits-per-byte metric
- `McEvaluator` — MMLU-style multiple-choice with per-subject breakdown
- `ExactMatchEvaluator` — text-match evaluation; `exact_match` / `f1_score` QA scoring
- ROUGE scoring: `RougeNScore` (ROUGE-1/2), `RougeLScore`, `RougeSScore`, `CorpusRouge`
- BLEU scoring: `BleuScore`, `sentence_bleu`, `corpus_bleu`
- ChrF (character n-gram F-score) metric
- METEOR metric
- Bootstrap confidence intervals for all metrics
- `ThroughputBenchmark` — tokens/s, prefill/decode latency, p95/p99
- `EvalDataset` — JSONL loading, train/test splits, deterministic sampling
- `EvalReport` — JSON and Markdown report generation
- Zero external API dependencies — pure Rust

## Usage

```toml
[dependencies]
pictor-eval = "0.2.2"
```

```rust
use pictor_eval::{PerplexityEvaluator, BleuScore};

// Perplexity from token log-probabilities
let log_probs = vec![-1.2, -0.8, -2.1, -1.5];
let ppl = PerplexityEvaluator::from_log_probs(&log_probs);
println!("Perplexity: {:.2}", ppl.perplexity());

// Corpus BLEU
let hypotheses = vec!["the cat sat on the mat".to_string()];
let references = vec![vec!["the cat is on the mat".to_string()]];
let bleu = BleuScore::corpus_bleu(&hypotheses, &references);
println!("BLEU: {:.4}", bleu.score());
```

## License

Apache-2.0 — derived from oxibonsai (COOLJAPAN OU). See [LICENSE](../../LICENSE) and [NOTICE](../../NOTICE).