[−][src]Trait syntaxdot_tokenizers::Tokenize

pub trait Tokenize: Send + Sync {
    pub fn tokenize(&self, sentence: Sentence) -> SentenceWithPieces;
}

Trait for wordpiece tokenizers.

Required methods

`pub fn tokenize(&self, sentence: Sentence) -> SentenceWithPieces`

Tokenize the tokens in a sentence into word pieces.

Loading content...

Implementors

`impl Tokenize for AlbertTokenizer`[src]

`pub fn tokenize(&self, sentence: Sentence) -> SentenceWithPieces`[src]

`impl Tokenize for BertTokenizer`[src]

`pub fn tokenize(&self, sentence: Sentence) -> SentenceWithPieces`[src]

`impl Tokenize for XlmRobertaTokenizer`[src]

`pub fn tokenize(&self, sentence: Sentence) -> SentenceWithPieces`[src]

Loading content...