Skip to main content

normalize

voxr::eval::normalize

Function normalize 

Source 
pub fn normalize(text: &str) -> Vec<String>
Expand description

Normalize text into comparison tokens.

Rules, applied in order:

  1. Lowercase (char::to_lowercase, so it is Unicode-aware, not ASCII-only).
  2. Every character that is neither alphanumeric nor ' becomes a space. This strips .,?!:;" and turns - and / into token boundaries.
  3. Leading and trailing ' are trimmed from each token, so 'word' normalizes to word while don't keeps its apostrophe.
  4. Split on whitespace; empty tokens are dropped.

Digits stay digits and number words stay words, so 250 and two hundred and fifty are four errors apart. Use crate::eval::canonicalize to score speech, where that difference is not a pronunciation difference.