Struct tantivy::tokenizer::TokenizerManager
source · [−]pub struct TokenizerManager { /* private fields */ }
Expand description
The tokenizer manager serves as a store for all of the pre-configured tokenizer pipelines.
By default, it is populated with the following managers.
raw
: does not process nor tokenize the text.default
: Chops the text on according to whitespace and punctuation, removes tokens that are too long, and lowercases tokensen_stem
: Likedefault
, but also applies stemming on the resulting tokens. Stemming can improve the recall of your search engine.
Implementations
sourceimpl TokenizerManager
impl TokenizerManager
sourcepub fn register<T>(&self, tokenizer_name: &str, tokenizer: T) where
TextAnalyzer: From<T>,
pub fn register<T>(&self, tokenizer_name: &str, tokenizer: T) where
TextAnalyzer: From<T>,
Registers a new tokenizer associated with a given name.
sourcepub fn get(&self, tokenizer_name: &str) -> Option<TextAnalyzer>
pub fn get(&self, tokenizer_name: &str) -> Option<TextAnalyzer>
Accessing a tokenizer given its name.
Trait Implementations
sourceimpl Clone for TokenizerManager
impl Clone for TokenizerManager
sourcefn clone(&self) -> TokenizerManager
fn clone(&self) -> TokenizerManager
Returns a copy of the value. Read more
1.0.0 · sourcefn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from source
. Read more
sourceimpl Default for TokenizerManager
impl Default for TokenizerManager
sourcefn default() -> TokenizerManager
fn default() -> TokenizerManager
Creates an TokenizerManager
prepopulated with
the default pre-configured tokenizers of tantivy
.
- simple
- en_stem
- ja
Auto Trait Implementations
impl RefUnwindSafe for TokenizerManager
impl Send for TokenizerManager
impl Sync for TokenizerManager
impl Unpin for TokenizerManager
impl UnwindSafe for TokenizerManager
Blanket Implementations
sourceimpl<T> BorrowMut<T> for T where
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
sourceimpl<T> Downcast for T where
T: Any,
impl<T> Downcast for T where
T: Any,
sourcefn into_any(self: Box<T, Global>) -> Box<dyn Any + 'static, Global>
fn into_any(self: Box<T, Global>) -> Box<dyn Any + 'static, Global>
Convert Box<dyn Trait>
(where Trait: Downcast
) to Box<dyn Any>
. Box<dyn Any>
can
then be further downcast
into Box<ConcreteType>
where ConcreteType
implements Trait
. Read more
sourcefn into_any_rc(self: Rc<T>) -> Rc<dyn Any + 'static>
fn into_any_rc(self: Rc<T>) -> Rc<dyn Any + 'static>
Convert Rc<Trait>
(where Trait: Downcast
) to Rc<Any>
. Rc<Any>
can then be
further downcast
into Rc<ConcreteType>
where ConcreteType
implements Trait
. Read more
sourcefn as_any(&self) -> &(dyn Any + 'static)
fn as_any(&self) -> &(dyn Any + 'static)
Convert &Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &Any
’s vtable from &Trait
’s. Read more
sourcefn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
Convert &mut Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &mut Any
’s vtable from &mut Trait
’s. Read more
sourceimpl<T> DowncastSync for T where
T: Any + Send + Sync,
impl<T> DowncastSync for T where
T: Any + Send + Sync,
impl<T> Pointable for T
impl<T> Pointable for T
sourceimpl<T> ToOwned for T where
T: Clone,
impl<T> ToOwned for T where
T: Clone,
type Owned = T
type Owned = T
The resulting type after obtaining ownership.
sourcefn clone_into(&self, target: &mut T)
fn clone_into(&self, target: &mut T)
toowned_clone_into
)Uses borrowed data to replace owned data, usually by cloning. Read more