Trait Tokenizer

pub trait Tokenizer:
    'static
    + Send
    + Sync
    + TokenizerClone {
    // Required method
    fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>;
}

Tokenizer are in charge of splitting text into a stream of token before indexing.

Warning

This API may change to use associated types.

Required Methods

fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>

Creates a token stream for a given str.

Implementors