Struct tantivy::tokenizer::TokenizerManager [−][src]
pub struct TokenizerManager { /* fields omitted */ }
Expand description
The tokenizer manager serves as a store for all of the pre-configured tokenizer pipelines.
By default, it is populated with the following managers.
raw
: does not process nor tokenize the text.default
: Chops the text on according to whitespace and punctuation, removes tokens that are too long, and lowercases tokensen_stem
: Likedefault
, but also applies stemming on the resulting tokens. Stemming can improve the recall of your search engine.
Implementations
Registers a new tokenizer associated with a given name.
Accessing a tokenizer given its name.
Trait Implementations
Creates an TokenizerManager
prepopulated with
the default pre-configured tokenizers of tantivy
.
- simple
- en_stem
- ja
Auto Trait Implementations
impl RefUnwindSafe for TokenizerManager
impl Send for TokenizerManager
impl Sync for TokenizerManager
impl Unpin for TokenizerManager
impl UnwindSafe for TokenizerManager
Blanket Implementations
Mutably borrows from an owned value. Read more
Convert Box<dyn Trait>
(where Trait: Downcast
) to Box<dyn Any>
. Box<dyn Any>
can
then be further downcast
into Box<ConcreteType>
where ConcreteType
implements Trait
. Read more
Convert Rc<Trait>
(where Trait: Downcast
) to Rc<Any>
. Rc<Any>
can then be
further downcast
into Rc<ConcreteType>
where ConcreteType
implements Trait
. Read more
Convert &Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &Any
’s vtable from &Trait
’s. Read more
Convert &mut Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &mut Any
’s vtable from &mut Trait
’s. Read more