Trait Tokenizer 

pub trait Tokenizer:
    'static
    + Clone
    + Send
    + Sync {
    type TokenStream<'a>: TokenStream;

    // Required method
    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>;
}

Tokenizer splits text into a token stream.

Required Associated Types

type TokenStream<'a>: TokenStream

Stream type emitted by the tokenizer.

Required Methods

fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>

Create a token stream for the provided text.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl Tokenizer for Box<dyn BoxableTokenizer>

type TokenStream<'a> = BoxTokenStream<'a>

fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>

Implementors

impl Tokenizer for NgramTokenizer

type TokenStream<'a> = NgramTokenStream<'a>

impl Tokenizer for RawTokenizer

type TokenStream<'a> = RawTokenStream<'a>

impl Tokenizer for SimpleTokenizer

type TokenStream<'a> = SimpleTokenStream<'a>

impl Tokenizer for WhitespaceTokenizer

type TokenStream<'a> = WhitespaceTokenStream<'a>