Module rust_tokenizers::tokenizer

Tokenizers

This module contains the tokenizers to split an input text in a sequence of tokens. These rely on the vocabularies for defining the subtokens a given word should be decomposed to. There are 3 main classes of tokenizers implemented in this crate:

• WordPiece tokenizers
  • BERT
  • DistilBERT
• Byte-Pair Encoding tokenizers:
  • GPT
  • GPT2
  • RoBERTa
  • CTRL
  • DeBERTa
• SentencePiece (Unigram) tokenizers:
  • SentencePiece
  • ALBERT
  • XLMRoBERTa
  • XLNet
  • T5
  • Marian
  • Reformer
  • DeBERTa (v2)

All tokenizers are Send, Sync and support multi-threaded tokenization and encoding.

Structs

• AlbertTokenizer
  ALBERT tokenizer
• BaseTokenizer
  Base tokenizer
• BertTokenizer
  BERT tokenizer
• CtrlTokenizer
  CTRL tokenizer
• DeBERTaTokenizer
  DeBERTa tokenizer
• DeBERTaV2Tokenizer
  DeBERTaV2 tokenizer
• FNetTokenizer
  FNet tokenizer
• Gpt2Tokenizer
  GPT2 tokenizer
• M2M100Tokenizer
  M2M100 tokenizer
• MBart50Tokenizer
  MBart50 tokenizer
• MarianTokenizer
  Marian tokenizer
• NLLBTokenizer
• OpenAiGptTokenizer
  GPT tokenizer
• PegasusTokenizer
  Pegasus tokenizer
• ProphetNetTokenizer
  ProphetNet tokenizer
• ReformerTokenizer
  Reformer tokenizer
• RobertaTokenizer
  RoBERTa tokenizer
• SentencePieceBpeTokenizer
  SentencePiece tokenizer
• SentencePieceTokenizer
  SentencePiece tokenizer
• T5Tokenizer
  T5 tokenizer
• XLMRobertaTokenizer
  XLM RoBERTa tokenizer
• XLNetTokenizer
  XLNet tokenizer

Enums

• TruncationStrategy
  Truncation strategy variants

Traits

• MultiThreadedTokenizer
  Extension for multithreaded tokenizers
• Tokenizer
  Base trait for tokenizers

Functions

• truncate_sequences
  Truncates a sequence pair in place to the maximum length.