Trait tantivy::tokenizer::Tokenizer

pub trait Tokenizer: 'static + Send + Sync + TokenizerClone {
    fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>;
}

Tokenizer are in charge of splitting text into a stream of token before indexing.

See the module documentation for more detail.

Warning

This API may change to use associated types.

Required methods

fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>

Creates a token stream for a given str.

Loading content...

Implementors

impl Tokenizer for FacetTokenizer

fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>

impl Tokenizer for NgramTokenizer

fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>

impl Tokenizer for RawTokenizer

fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>

impl Tokenizer for SimpleTokenizer

fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>

Loading content...