pub fn normalize_input(x: &str) -> Cow<'_, str>Expand description
The input clean-up every parse starts with: normalize_spaces, invisible format characters
removed — zero-width space and joiners U+200B–U+200D, the word joiner U+2060 and the byte order
mark U+FEFF (the first cell of a UTF-8 file with a BOM) — and the fullwidth ASCII forms
U+FF01–U+FF5E folded to ASCII, as East Asian keyboards type them ((Tubangui,1928)). The
homoglyph table cannot do the latter: it maps fullwidth l and I to the digit 1. The soft
hyphen U+00AD only marks where a word may break: it is dropped (novae\u{ad}zelandiae,
Ramí\u{ad}rez), except between a lower-case and a capital letter, where it stands for a real
hyphen (Miranda\u{ad}Ribeiro). Mojibake is repaired first (repair_mojibake): its
continuation bytes include the soft hyphen and the no-break space (Ã\u{ad} is í, Ã\u{a0}
is à). Decomposed letters are composed (NFC): a combining accent after its letter
(Za\u{301}gors\u{30c}ek) otherwise splits the word, and Java read Za gors ek. Borrows when
there is nothing to change.