ffai_bench/lib.rs
1//! # ffai-bench — `FFai`'s analyzer
2//!
3//! One call to answer: **how does our engine compare to the world standard,
4//! on pinned data, reproducibly?** `ffai bench asr --corpus corpora/x.toml`
5//! runs our engine and any configured reference implementations over the same
6//! holdout clips, computes task metrics (WER/CER, real-time factor), and
7//! appends an audit-grade record to the claims ledger.
8//!
9//! ## Lineage
10//!
11//! The measurement spine is ported from **Prometheus**, the private refinery
12//! built for `remade_ffmpeg_rs`: the four-gate verdict, best-of-N wall-clock
13//! timing, hashed corpus manifests with clip-level train/holdout splits, and
14//! the append-only JSONL ledger where a skipped gate is never a pass and
15//! losses are recorded as knowledge. The symbolic-discovery half of Prometheus
16//! (symreg → E-graph simplify → SMT prove → codegen) stays private and
17//! codec-focused; it does not apply to learned-model engines.
18//!
19//! This crate is public on purpose: performance/quality claims `FFai` makes are
20//! only worth making if anyone can re-run them from a ledger line alone.
21//!
22//! ## The four gates (adapted from Prometheus for model engines)
23//!
24//! | Gate | Prometheus meaning | FFai meaning |
25//! |---|---|---|
26//! | correctness | bit-exact / conformance | engine completes every holdout clip with well-formed output |
27//! | quality | corpus BD-rate / PEAQ | task metric vs reference (WER/CER parity band) on holdout |
28//! | speed | best-of-N vs cycle budget | best-of-N real-time factor vs reference |
29//! | footprint | SMT safety proof | peak memory / binary size budget (instrumented in Phase 1) |
30
31pub mod corpus;
32pub mod der;
33pub mod detect;
34pub mod footprint;
35pub mod gate;
36pub mod ledger;
37pub mod metrics;
38pub mod normalize;
39pub mod reference;
40pub mod resample;
41pub mod runner;
42pub mod speed;
43pub mod tts;
44pub mod vlm;