Skip to main content

Crate lance_tokenizer

Crate lance_tokenizer 

Source

Structs§

AlphaNumOnlyFilter
AsciiFoldingFilter
BoxTokenStream
Token stream object-safe wrapper.
LowerCaser
NgramTokenizer
RawTokenizer
RemoveLongFilter
SimpleTokenStream
SimpleTokenizer
Stemmer
StopWordFilter
TextAnalyzer
TextAnalyzerBuilder
Token
Token emitted by a tokenizer.
WhitespaceTokenizer

Enums§

Language

Traits§

TokenFilter
Filter that wraps a tokenizer with additional token-processing behavior.
TokenStream
Consumable token stream.
Tokenizer
Tokenizer splits text into a token stream.