Crate syntaxdot_tokenizers

Source

Structs§

AlbertTokenizer
Tokenizer for ALBERT models.
BertTokenizer
BERT word piece tokenizer.
SentenceWithPieces
A sentence and its word pieces.
XlmRobertaTokenizer
Tokenizer for Roberta models.

Enums§

TokenizerError

Traits§

Tokenize
Trait for wordpiece tokenizers.