Expand description
Text normalization for error-rate scoring — a port of OpenAI Whisper’s
whisper/normalizers/{basic,english}.py.
§Why this exists
Raw WER between an ASR hypothesis and a LibriSpeech reference is mostly
noise about formatting. The reference says MISTER QUILTER, Whisper
writes Mr. Quilter; the reference says TWENTY THREE, Whisper writes
23. Scoring those as errors would make every implementation — ours and
the references alike — look far worse than it is, and would put our
published numbers nowhere near the world’s.
Normalization is applied identically to reference and hypothesis, and identically to every implementation under test, so it cannot advantage anyone.
§Parity status (honest accounting)
Implemented faithfully: lowercasing, bracket/parenthesis stripping, filler removal, the contraction and title replacer table, comma-in-digit and period handling, symbol stripping with the numeric keep-set, and spelled-number → digit conversion.
Not yet at bit-parity with openai-whisper, tracked as a Mercury M1 exit item:
- the ~1,700-entry British→American spelling map (
english.json), - Unicode NFKD decomposition and general-category-based diacritic removal (we approximate: non-alphanumeric, non-keep characters become spaces),
- fractions, currency-suffix forms, and the year-pair heuristics in
EnglishNumberNormalizer.
Until those land, treat cross-implementation comparisons produced here as sound (same normalizer for everyone) and absolute agreement with published WER figures as approximate.
Enums§
- Mode
- Which normalizer to score with.
Functions§
- normalize
- Normalize
textundermode.