Module tokenizer 

The core tokenizer, v2.

The pipeline mirrors what the strongest lexical engines (Lucene’s ICU/Standard analyzers, SQLite FTS5 unicode61) converge on, built from the pure-core unicode-rs table crates so it runs identically on native and every wasm runtime:

1. NFKC normalization of the input (fullwidth Ａ → A, ligature fi → fi, decomposed marks recomposed) — one pass into a reused scratch buffer.
2. UAX #29 word segmentation (unicode-segmentation) — the same boundary standard ICU implements. Consequences worth knowing: don't and o'clock stay whole (apostrophe joins letters), 3.14 / v1.2.3 / example.com stay whole (./, join digits and letters), snake_case stays whole (_ joins), gpt-4o splits on the hyphen.
3. Per-token folding: full Unicode lowercase; Latin diacritics stripped (café → cafe, the FTS5 remove_diacritics behavior, applied only to Latin bases so Cyrillic й is untouched); the Russian-specific ё → е fold every major Russian search engine applies.
4. CJK: Han ideographs and Hiragana come out of UAX #29 as single-character segments; adjacent ones are joined into overlapping bigrams (the Lucene CJKBigramFilter scheme — the standard dictionary-free CJK treatment), a lone character stays a unigram. Katakana and Hangul already segment into word runs and are kept as words.
5. A token longer than MAX_TOKEN_BYTES is truncated at the last char boundary that fits (long tokens sharing a 64-byte prefix collapse — accepted by spec).

No stemming or lemmatization in v1. Emitted tokens are canonical: a fixed point of the tokenizer (fixed by a property test).

Structs

Tokenizer

Streaming tokenizer with reusable scratch buffers.

Constants

MAX_TOKEN_BYTES

Upper bound on an emitted token, in bytes.