Module analyzer 

Composable text analysis pipeline.

text -> CharFilter* -> Tokenizer -> TokenFilter* -> tokens

Structs

Analyzer

Functions

keyword_analyzer

Keyword tokenizer with no filters.

standard_analyzer

Standard + Lowercase + ASCIIFolding + Stop + PorterStem.

standard_cjk_analyzer

standard_analyzer extended with character-level n-grams (2..=3) for CJK-style text where words are not whitespace-delimited.

whitespace_analyzer

WhitespaceTokenizer + Lowercase.