Expand description
Semantic Tokenizer — text tokenization for semantic search indexing.
Provides configurable tokenization with whitespace, word boundary, and n-gram modes, stop-word filtering, length constraints, and normalization.
Structs§
- Semantic
Tokenizer - A configurable tokenizer for semantic search indexing.
- Token
- A single token produced by the tokenizer.
- Tokenizer
Config - Configuration for
SemanticTokenizer. - Tokenizer
Stats - Runtime statistics for a
SemanticTokenizer.
Enums§
- Tokenizer
Mode - Tokenization strategy.