Crate ffai_bench 

ffai-bench — FFai’s analyzer

One call to answer: how does our engine compare to the world standard, on pinned data, reproducibly? ffai bench asr --corpus corpora/x.toml runs our engine and any configured reference implementations over the same holdout clips, computes task metrics (WER/CER, real-time factor), and appends an audit-grade record to the claims ledger.

Lineage

The measurement spine is ported from Prometheus, the private refinery built for remade_ffmpeg_rs: the four-gate verdict, best-of-N wall-clock timing, hashed corpus manifests with clip-level train/holdout splits, and the append-only JSONL ledger where a skipped gate is never a pass and losses are recorded as knowledge. The symbolic-discovery half of Prometheus (symreg → E-graph simplify → SMT prove → codegen) stays private and codec-focused; it does not apply to learned-model engines.

This crate is public on purpose: performance/quality claims FFai makes are only worth making if anyone can re-run them from a ledger line alone.

The four gates (adapted from Prometheus for model engines)

Gate	Prometheus meaning	FFai meaning
correctness	bit-exact / conformance	engine completes every holdout clip with well-formed output
quality	corpus BD-rate / PEAQ	task metric vs reference (WER/CER parity band) on holdout
speed	best-of-N vs cycle budget	best-of-N real-time factor vs reference
footprint	SMT safety proof	peak memory / binary size budget (instrumented in Phase 1)

Modules

corpus

Hashed corpus manifests, ported from Prometheus (prom-corpus).

der

Diarization Error Rate — the metric that gates --diarize.

detect

Detection scoring: a COCO-style mAP proxy, beside CER/WER as der.rs is beside them for diarization.

footprint

Peak-memory measurement — the fourth gate’s instrument.

gate

The four-gate verdict, ported from Prometheus (prom-core::gate). Nothing is claimed unless all four pass; a skipped gate is never a pass.

ledger

The claims ledger, ported from Prometheus (prom-ledger).

metrics

Task metrics: WER and CER (the ASR/OCR standards).

normalize

Text normalization for error-rate scoring — a port of OpenAI Whisper’s whisper/normalizers/{basic,english}.py.

reference

World-standard reference adapters — the “oracle” seat from Prometheus’s trial stage (prom-trial::oracle), generalized to external AI tools.

resample

Harness-side resampling: every TTS implementation’s audio is converted to the judge’s required format (16 kHz mono) by THIS code, identically, so no implementation’s score depends on its native sample rate or on some external tool’s resampler. This is measurement plumbing, not product — engines still reject wrong-rate audio rather than silently resampling (ffai-media’s documented policy).

runner

The one-call bench: our engine vs world standards on a pinned corpus.

speed

Best-of-N wall-clock measurement, ported from Prometheus (prom-trial::speed). We report the best (minimum) time — the run least perturbed by scheduler noise — plus the median for a spread sense.

tts

The TTS bench vertical: synthesize a pinned text corpus, judge every implementation’s audio with the same frozen third-party ASR, score round-trip WER/CER against the input text, and append the record to the ledger. Mission plan: docs/mercury-tts-mission.md §5.

vlm

The VLM bench vertical (Argus) — predictions are ours, scoring is theirs.