Struct tantivy::tokenizer::TokenizerManager

pub struct TokenizerManager { /* fields omitted */ }

The tokenizer manager serves as a store for all of the pre-configured tokenizer pipelines.

By default, it is populated with the following managers.

• raw : does not process nor tokenize the text.
• default : Chops the text on according to whitespace and punctuation, removes tokens that are too long, and lowercases tokens
• en_stem : Like default, but also applies stemming on the resulting tokens. Stemming can improve the recall of your search engine.

Methods

impl TokenizerManager

pub fn register<A>(&self, tokenizer_name: &str, tokenizer: A) where
    A: 'static + Send + Sync + for<'a> Tokenizer<'a>, 

Registers a new tokenizer associated with a given name.

pub fn get(&self, tokenizer_name: &str) -> Option<Box<BoxedTokenizer>>

Accessing a tokenizer given its name.

Trait Implementations

impl Clone for TokenizerManager

fn clone(&self) -> TokenizerManager

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for TokenizerManager

fn default() -> TokenizerManager

Creates an TokenizerManager prepopulated with the default pre-configured tokenizers of tantivy.

• simple
• en_stem
• ja