Module tokenizer

Expand description

Tokenizer API for text processing

Modules§

HfTokenizer: Cached HuggingFace tokenizer
IdfWeights: Pre-computed IDF weights indexed by token_id
IdfWeightsCache: Global cache for IDF weights, keyed by model name. Caches both successful loads and failures to avoid repeated download attempts.
LanguageAwareTokenizer: Language-aware tokenizer that can be configured per-field
LexOptions: Options of a LexTokenizer; the parsed and rendered form of a lex(...) spec. Only values that differ from the defaults are rendered.
LexTokenizer: The tokenizer of a lex(...) field (module docs).
MultiLanguageStemmer: Multi-language stemmer that can select language dynamically
RawCiTokenizer: Raw case-insensitive tokenizer — lowercases the entire input without splitting.
RawTokenizer: Raw tokenizer — no tokenization at all.
SimpleTokenizer: Simple tokenizer — splits on whitespace, strips non-alphanumeric, and lowercases.
StemmerTokenizer: Stemming tokenizer - splits on whitespace, lowercases, and applies stemming
StopWordTokenizer: Stop word filter tokenizer - wraps another tokenizer and filters out stop words
Token: A token produced by tokenization
TokenizerCache: Global tokenizer cache for reuse across queries
TokenizerRegistry: Registry for named tokenizers
UnicodeWordTokenizer: Unicode word boundaries followed by lowercase, without lexical rewriting. Internal punctuation follows UAX #29. Overlong words retain a position gap.

CjkMode: Japanese and Korean analysis.
HanForm: Treatment of Han characters.
Language: Supported stemmer languages
Purpose: What a tokenization is for.
Script: Writing system of a token, used to route it to a stemmer of the same script.
Segmenter: Word segmentation.
StemMode: Morphological normalisation.
TokenizerSource: Tokenizer source - where to load the tokenizer from
TokenizerSpec: Parsed form of a tokenizer name in the schema: a registered name (simple, en_stem, …) or a lex(...) spec. The canonical string form is stored in FieldEntry::tokenizer.

DEFAULT_MAX_TOKEN_LENGTH: Default max_token_length: longer “words” are hashes, sequences and URLs, which no query types and which bloat the dictionary.

cjk_dictionaries_available: Whether the dictionaries are compiled in.
idf_weights_cache: Get the global IDF weights cache
language_code: ISO 639-1 code of a stemmer language.
parse_language: Parse a language string into a Language enum
parse_language_opt: Parse a language string into a Language, returning None for unknown values.
tokenizer_cache: Get the global tokenizer cache
warm_up_cjk_dictionaries: Load the dictionaries now (a server calls this at start-up so the first Japanese or Korean request does not pay the decode).