pub fn normalize_input(x: &str) -> Cow<'_, str>Expand description
The input clean-up every parse starts with: normalize_spaces, invisible format characters
removed — zero-width space and joiners U+200B–U+200D, the word joiner U+2060 and the byte order
mark U+FEFF (the first cell of a UTF-8 file with a BOM) — and the fullwidth ASCII forms
U+FF01–U+FF5E folded to ASCII, as East Asian keyboards type them ((Tubangui,1928)). The
homoglyph table cannot do the latter: it maps fullwidth l and I to the digit 1. The soft
hyphen U+00AD only marks where a word may break: it is dropped (novae\u{ad}zelandiae,
Ramí\u{ad}rez), except between a lower-case and a capital letter, where it stands for a real
hyphen (Miranda\u{ad}Ribeiro), and after Ã, where it is half of the Windows-1252 mojibake
Ã\u{ad} for í, repaired later. Decomposed letters are composed (NFC): a combining accent
after its letter (Za\u{301}gors\u{30c}ek) otherwise splits the word, and Java read
Za gors ek. Borrows when there is nothing to change.