Skip to main content

Module tokenizer

Module tokenizer 

Source

Structs§

Tokenizer
Unicode-aware tokenizer with configurable pipeline: split → lowercase → clean → length filter → stopword filter → suffix strip.
TokenizerConfig
Configuration for the tokenizer pipeline.