pub fn normalize_input(x: &str) -> Cow<'_, str>Expand description
The input clean-up every parse starts with: normalize_spaces, invisible format characters
removed — zero-width space and joiners U+200B–U+200D, the word joiner U+2060 and the byte order
mark U+FEFF (the first cell of a UTF-8 file with a BOM) — and the fullwidth ASCII forms
U+FF01–U+FF5E folded to ASCII, as East Asian keyboards type them ((Tubangui,1928)). The
homoglyph table cannot do the latter: it maps fullwidth l and I to the digit 1. Not the
soft hyphen U+00AD: in the data it is mostly half of the Windows-1252 mojibake Ã\u{ad} for
í, repaired later, or a stand-in for a real hyphen (Miranda\u{ad}Ribeiro). Decomposed
letters are composed (NFC): a combining accent after its letter (Za\u{301}gors\u{30c}ek)
otherwise splits the word, and Java read Za gors ek. Borrows when there is nothing to change.