List of all items

Structs

• CharsMap
• Configuration
• Definition
• Kitoken
• Metadata
• Regex
• RegexError
• SpecialToken
• Template
• Token

Enums

• ConfigurationError
• DecodeError
• Decoding
• DecodingReplacePattern
• DeserializationError
• EncodeError
• Fallback
• InitializationError
• InsertionPosition
• Model
• Normalization
• NormalizationCondition
• NormalizationReplacePattern
• Processing
• ProcessingDirection
• SpecialTokenKind
• Split
• SplitBehavior
• SplitPattern
• UnicodeNormalization
• convert::ConversionError

Functions

• convert::convert_sentencepiece
• convert::convert_tekken
• convert::convert_tiktoken
• convert::convert_tokenizers

Type Aliases

• Scores
• SpecialTokenIdent
• SpecialVocab
• TokenBytes
• TokenId
• TokenScore
• Vocab