Expand description
Versioned quality evaluation helpers (JOE-1607).
Offline, deterministic metrics for STT (WER/CER) and simple TTS objective
checks. Fixture corpora live under evals/ at the repo root.
Structs§
- Eval
Corpus - Corpus manifest (versioned).
- Eval
Report - Aggregate report (machine-readable).
- SttFixture
- One STT reference fixture (text-only; audio path is optional for offline text scoring).
- SttScore
- STT scoring result for one hypothesis.
- TtsFixture
- TTS text fixture for pronunciation / chunk-join checks.
Functions§
- build_
report - Build an aggregate report from scores.
- char_
error_ rate - Character error rate on normalized strings (spaces kept as characters).
- normalize_
transcript - Normalize text for WER: lowercase, strip punctuation, collapse whitespace.
- repetition_
ratio - Degeneration proxy: longest run of identical tokens over hypothesis length.
- score_
stt - Score one STT hypothesis against a fixture.
- silence_
false_ positive - Silence false positive: non-empty hypothesis for an empty reference.
- smoke_
corpus - Built-in smoke corpus (synthetic text + optional synthetic audio paths).
- tts_
duration_ in_ range - Objective TTS duration check (not MOS).
- word_
error_ rate - Word error rate via classic Levenshtein on word tokens.