Skip to main content

normalize_key

Function normalize_key 

Source
pub fn normalize_key(s: &str) -> String
Expand description

cmf-key-v2 (KEY_NORM): NFC → every combining mark and every invisible format character is dropped WITHOUT ending the word (a stress mark over a Cyrillic vowel has no precomposed form, so NFC leaves U+0301 standing: Рома́шка → ромашка; soft hyphen, zero-width space / joiner / non-joiner, word joiner, BOM: ро­машка → ромашка) → a precomposed LATIN letter loses its diacritics (Pínus → pinus, école → ecole; a Cyrillic letter keeps its mark — й, ё, ў are letters, not accents) → lowercase → every other char that is not alphanumeric (Unicode) becomes a space → runs of spaces collapse → trim. Shared by the builder (keys) and the runtime (n-grams of the user message), so normalize_key("Пихта бальзамическая (Abies balsamea)") == "пихта бальзамическая abies balsamea". Idempotent.