Trait Tokenizer 

pub trait Tokenizer {
    // Required method
    fn tokenize<'a>(
        &'a self,
        text: &'a str,
    ) -> JPreprocessResult<Vec<impl 'a + Token>>;
}

Required Methods

fn tokenize<'a>( &'a self, text: &'a str, ) -> JPreprocessResult<Vec<impl 'a + Token>>

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl Tokenizer for Tokenizer

Available on crate feature lindera only.

fn tokenize<'a>( &'a self, text: &'a str, ) -> JPreprocessResult<Vec<impl 'a + Token>>

impl Tokenizer for Tokenizer

Available on crate feature vibrato only.

Vibrato support is experimental and may be removed or changed in the future. Use with caution.

fn tokenize<'a>( &'a self, text: &'a str, ) -> JPreprocessResult<Vec<impl 'a + Token>>

Implementors