Trait tokenizer::Tokenizer

pub trait Tokenizer {
    fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>;
}

A trait that all Tokenizer should implement.

Required methods

fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>

Tokenize given text and return a Vec<&str> where each &str inside a Vec is a slice from given text.

Loading content...

Implementors

impl Tokenizer for tokenizer::en::Tokenizer

fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>

impl Tokenizer for tokenizer::th::Tokenizer

fn tokenize<'b>(&self, value: &'b str) -> Vec<&'b str>

Loading content...