List of all items[−]
Structs
- ConsolidatedTokenIterator
- Offset
- Token
- TokenIdsWithOffsets
- TokenIdsWithSpecialTokens
- TokenRef
- TokenizedInput
- TokensWithOffsets
- adapters::Example
- tokenizer::AlbertTokenizer
- tokenizer::BaseTokenizer
- tokenizer::BertTokenizer
- tokenizer::CtrlTokenizer
- tokenizer::Gpt2Tokenizer
- tokenizer::MBart50Tokenizer
- tokenizer::MarianTokenizer
- tokenizer::OpenAiGptTokenizer
- tokenizer::PegasusTokenizer
- tokenizer::ProphetNetTokenizer
- tokenizer::ReformerTokenizer
- tokenizer::RobertaTokenizer
- tokenizer::SentencePieceTokenizer
- tokenizer::T5Tokenizer
- tokenizer::XLMRobertaTokenizer
- tokenizer::XLNetTokenizer
- vocab::AlbertVocab
- vocab::BaseVocab
- vocab::BertVocab
- vocab::BpePairRef
- vocab::BpePairVocab
- vocab::Gpt2Vocab
- vocab::MBart50Vocab
- vocab::MarianVocab
- vocab::OpenAiGptVocab
- vocab::PegasusVocab
- vocab::ProphetNetVocab
- vocab::ReformerVocab
- vocab::RobertaVocab
- vocab::SentencePieceModel
- vocab::SentencePieceVocab
- vocab::T5Vocab
- vocab::XLMRobertaVocab
- vocab::XLNetVocab