Trait Tokenizer 

pub trait Tokenizer {
    // Required method
    fn tokenize(&self, text: &str) -> Vec<String>;
}

A trait for tokenizing text. This trait is implemented by the BasicTokenizer and WordPieceTokenizer.

Required Methods

fn tokenize(&self, text: &str) -> Vec<String>

Implementors

impl Tokenizer for BasicTokenizer

impl Tokenizer for FullTokenizer

impl Tokenizer for WordPieceTokenizer