Crate syntaxdot_tokenizers

Structs

• AlbertTokenizer
  Tokenizer for ALBERT models.
• BertTokenizer
  BERT word piece tokenizer.
• SentenceWithPieces
  A sentence and its word pieces.
• XlmRobertaTokenizer
  Tokenizer for Roberta models.

Enums

• TokenizerError

Traits

• Tokenize
  Trait for wordpiece tokenizers.