Trait Tokenizer 

pub trait Tokenizer {
    // Required method
    fn tokenize<'a>(
        &'a self,
        text: &'a str,
    ) -> JPreprocessResult<Vec<impl 'a + Token>>;
}

Required Methods

fn tokenize<'a>( &'a self, text: &'a str, ) -> JPreprocessResult<Vec<impl 'a + Token>>

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementations on Foreign Types

impl Tokenizer for Tokenizer

fn tokenize<'a>( &'a self, text: &'a str, ) -> JPreprocessResult<Vec<impl 'a + Token>>

Implementors

impl Tokenizer for DefaultTokenizer

impl Tokenizer for JPreprocessTokenizer