Trait Tokenizer 

pub trait Tokenizer:
    Send
    + Sync
    + Clone
    + 'static {
    // Required method
    fn tokenize(&self, text: &str) -> Vec<Token>;

    // Provided method
    fn tokenize_with(
        &self,
        text: &str,
        hint: Option<&str>,
        purpose: Purpose,
    ) -> Vec<Token> { ... }
}

Trait for tokenizers

Required Methods

fn tokenize(&self, text: &str) -> Vec<Token>

Tokenize the input text into a vector of tokens

Provided Methods

fn tokenize_with( &self, text: &str, hint: Option<&str>, purpose: Purpose, ) -> Vec<Token>

Tokenize with a caller-supplied hint and a purpose.

Static tokenizers ignore both and have a single form. A LexTokenizer reads the hint as a comma-separated list of language codes (a sibling document field at index time, tokenizer_hint on the query) and emits the forms the purpose asks for (see Purpose).

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl Tokenizer for LexTokenizer

impl Tokenizer for MultiLanguageStemmer

impl Tokenizer for RawCiTokenizer

impl Tokenizer for RawTokenizer

impl Tokenizer for SimpleTokenizer

impl Tokenizer for StemmerTokenizer

impl Tokenizer for UnicodeWordTokenizer

impl<F> Tokenizer for LanguageAwareTokenizer<F>

where F: Fn(&str) -> Language + Clone + Send + Sync + 'static,

impl<T: Tokenizer> Tokenizer for StopWordTokenizer<T>