[−][src]Trait tantivy::tokenizer::Tokenizer
Tokenizer
are in charge of splitting text into a stream of token
before indexing.
See the module documentation for more detail.
Warning
This API may change to use associated types.
Required methods
fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>
Creates a token stream for a given str
.