Module tokenizer 

Tokenizer API for text processing

Structs

LanguageAwareTokenizer

Language-aware tokenizer that can be configured per-field

LowercaseTokenizer

Lowercase tokenizer - splits on whitespace and lowercases

MultiLanguageStemmer

Multi-language stemmer that can select language dynamically

SimpleTokenizer

Simple whitespace tokenizer

StemmerTokenizer

Stemming tokenizer - splits on whitespace, lowercases, and applies stemming

StopWordTokenizer

Stop word filter tokenizer - wraps another tokenizer and filters out stop words

Token

A token produced by tokenization

TokenizerRegistry

Registry for named tokenizers

Enums

Language

Supported stemmer languages

Traits

Tokenizer

Trait for tokenizers

TokenizerClone

Functions

parse_language

Parse a language string into a Language enum

Type Aliases

BoxedTokenizer

Boxed tokenizer for dynamic dispatch