Struct tantivy::tokenizer::TokenizerManager
source · pub struct TokenizerManager { /* private fields */ }
Expand description
The tokenizer manager serves as a store for all of the pre-configured tokenizer pipelines.
By default, it is populated with the following managers.
raw
: does not process nor tokenize the text.default
: Chops the text on according to whitespace and punctuation, removes tokens that are too long, and lowercases tokensen_stem
: Likedefault
, but also applies stemming on the resulting tokens. Stemming can improve the recall of your search engine.whitespace
: Splits the text on whitespaces.
Implementations§
source§impl TokenizerManager
impl TokenizerManager
sourcepub fn register<T>(&self, tokenizer_name: &str, tokenizer: T)where
TextAnalyzer: From<T>,
pub fn register<T>(&self, tokenizer_name: &str, tokenizer: T)where
TextAnalyzer: From<T>,
Registers a new tokenizer associated with a given name.
sourcepub fn get(&self, tokenizer_name: &str) -> Option<TextAnalyzer>
pub fn get(&self, tokenizer_name: &str) -> Option<TextAnalyzer>
Accessing a tokenizer given its name.
Trait Implementations§
source§impl Clone for TokenizerManager
impl Clone for TokenizerManager
source§fn clone(&self) -> TokenizerManager
fn clone(&self) -> TokenizerManager
Returns a copy of the value. Read more
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source
. Read moresource§impl Default for TokenizerManager
impl Default for TokenizerManager
source§fn default() -> TokenizerManager
fn default() -> TokenizerManager
Creates an TokenizerManager
prepopulated with
the default pre-configured tokenizers of tantivy
.
Auto Trait Implementations§
impl Freeze for TokenizerManager
impl RefUnwindSafe for TokenizerManager
impl Send for TokenizerManager
impl Sync for TokenizerManager
impl Unpin for TokenizerManager
impl UnwindSafe for TokenizerManager
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
source§impl<T> Downcast for Twhere
T: Any,
impl<T> Downcast for Twhere
T: Any,
source§fn into_any(self: Box<T>) -> Box<dyn Any>
fn into_any(self: Box<T>) -> Box<dyn Any>
Convert
Box<dyn Trait>
(where Trait: Downcast
) to Box<dyn Any>
. Box<dyn Any>
can
then be further downcast
into Box<ConcreteType>
where ConcreteType
implements Trait
.source§fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>
fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>
Convert
Rc<Trait>
(where Trait: Downcast
) to Rc<Any>
. Rc<Any>
can then be
further downcast
into Rc<ConcreteType>
where ConcreteType
implements Trait
.source§fn as_any(&self) -> &(dyn Any + 'static)
fn as_any(&self) -> &(dyn Any + 'static)
Convert
&Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &Any
’s vtable from &Trait
’s.source§fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
Convert
&mut Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &mut Any
’s vtable from &mut Trait
’s.