[−][src]Trait tantivy::tokenizer::BoxedTokenizer

pub trait BoxedTokenizer: Send + Sync {
    fn token_stream<'a>(&self, text: &'a str) -> Box<dyn TokenStream + 'a>;
    fn token_stream_texts<'b>(
        &self, 
        texts: &'b [&'b str]
    ) -> Box<dyn TokenStream + 'b>;
    fn boxed_clone(&self) -> Box<dyn BoxedTokenizer>;
}

A boxed tokenizer

Required methods

`fn token_stream<'a>(&self, text: &'a str) -> Box<dyn TokenStream + 'a>`

Tokenize a &str

`fn token_stream_texts<'b>( &self, texts: &'b [&'b str] ) -> Box<dyn TokenStream + 'b>`

Tokenize an array&str

The resulting TokenStream is equivalent to what would be obtained if the &str were one concatenated &str, with an artificial position gap of 2 between the different fields to prevent accidental PhraseQuery to match accross two terms.

`fn boxed_clone(&self) -> Box<dyn BoxedTokenizer>`

Return a boxed clone of the tokenizer

Loading content...

Implementors