Module text_analyzer 

Text analysis pipeline for full-text search indexing and querying.

Pipeline stages (applied at both index time and query time):

1. Unicode NFD normalization + strip combining marks
2. Lowercase
3. Split on non-alphanumeric boundaries (preserving hyphens within words)
4. Filter empty tokens and single characters
5. Remove English stop words (binary search on sorted list)
6. Snowball English stemming

Produces a Vec<String> of normalized, stemmed tokens suitable for inverted index insertion and BM25 query matching.

Structs

AnalyzerRegistry

Registry of named text analyzers and synonym maps per collection.

EdgeNgramAnalyzer

Edge n-gram analyzer: generates n-grams anchored to the start of each token.

KeywordAnalyzer

Keyword analyzer: treats entire input as a single token (lowercase).

LanguageAnalyzer

Language-specific analyzer: uses Snowball stemming for the configured language.

NgramAnalyzer

N-gram analyzer: generates all character n-grams of sizes min..=max for each token.

SimpleAnalyzer

Simple analyzer: lowercase + split on whitespace. No stemming or stop words.

StandardAnalyzer

Standard English text analyzer (default).

SynonymMap

Synonym map: expands query terms with their synonyms at query time.

Traits

TextAnalyzer

Text analyzer trait: transforms raw text into searchable tokens.

Functions

analyze

Analyze text into searchable tokens using the standard English analyzer.