Trait bert_tokenizer::Tokenizer
source · pub trait Tokenizer {
// Required method
fn tokenize(&self, text: &str) -> Vec<String>;
}
Expand description
A trait for tokenizing text.
This trait is implemented by the BasicTokenizer
and WordPieceTokenizer
.