pub fn words(text: &str) -> impl Iterator<Item = Word<'_>>Expand description
The words in text, in order. A word is a run of letters, with any combining marks on them, so
accents, Indic vowel signs, and Arabic vowel marks stay in their word. An apostrophe (' or ’), the
Catalan middle dot, Hebrew’s geresh and gershayim, ZWNJ, ZWJ, or a soft hyphen between two letters
stays inside too, so “don’t”, “l’arbre”, and “col·lecció” are one word each. Digits, hyphens, and
everything else end a word.
Chinese and Japanese put no spaces between words, so each of their characters (Han, kana, and Bopomofo) is a word of its own. Thai, Lao, Khmer, and Burmese don’t either, but there each run of letters is one word.
Letters come from a table of Unicode 17, the same one the Kotlin kit’s words and the C kit’s
hammer_words use, so all three split text the same way.