[−][src]Trait tantivy::tokenizer::BoxedTokenizer
A boxed tokenizer
Required methods
fn token_stream<'a>(&self, text: &'a str) -> Box<dyn TokenStream + 'a>
Tokenize a &str
fn token_stream_texts<'b>(
&self,
texts: &'b [&'b str]
) -> Box<dyn TokenStream + 'b>
&self,
texts: &'b [&'b str]
) -> Box<dyn TokenStream + 'b>
Tokenize an array&str
The resulting TokenStream
is equivalent to what would be obtained if the &str were
one concatenated &str
, with an artificial position gap of 2
between the different fields
to prevent accidental PhraseQuery
to match accross two terms.
fn boxed_clone(&self) -> Box<dyn BoxedTokenizer>
Return a boxed clone of the tokenizer