Module tokenizer 

The core tokenizer, v2.

The tokenizer is deliberately split into four stages:

1. NFKC normalization of the input;
2. UAX #29 word segmentation and CJK adjacency tracking;
3. policy-driven Unicode folding;
4. canonical, byte-budgeted token emission.

The stages share caller-owned scratch buffers, so the ordinary Latin, Cyrillic, and generic-script paths remain allocation-free after warm-up and behave identically on native and WASM. ICU4X’s dictionary/LSTM path for complex scripts may allocate inside its iterator, but uses the same token policy and canonical emission rules. Emitted tokens are canonical fixed points of the tokenizer.

Structs

Tokenizer

Streaming tokenizer with reusable scratch buffers.

TokenizerPolicy

Controls search-specific folding without changing the scanner or its allocation behavior.

Constants

MAX_TOKEN_BYTES

Upper bound on an emitted token, in bytes.