Skip to main content

Tokenizer

jpreprocess_core::token

Trait Tokenizer 

Source 
pub trait Tokenizer {
    // Required method
    fn tokenize<'a>(
        &'a self,
        text: &'a str,
    ) -> JPreprocessResult<Vec<impl 'a + Token>>;
}

Required Methods§

Source

fn tokenize<'a>( &'a self, text: &'a str, ) -> JPreprocessResult<Vec<impl 'a + Token>>

Dyn Compatibility§

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§