Module tokenizer 

Text tokenization and stopword filtering.

Structs

Tokenizer

A configured tokenizer that normalizes and filters text into terms.

TokenizerConfig

Configuration for the tokenizer.