List of all items

Structs

• DecodeStream
• Error
• Sequence
• StopSequenceDecoder
• StopSequenceDecoderBuilder
• Tokenizer
• TokenizerOptions
• cache::CacheTokenUsage
• cache::CachedTokenizer
• cache::L1Cache
• cache::L1CacheStats
• fastokens::FastTokenizer
• hf::HuggingFaceTokenizer
• tiktoken::TikTokenTokenizer

Enums

• Encoding
• SequenceDecoderOutput
• TokenizerType
• traits::DecodeResult

Traits

• traits::Decoder
• traits::Encoder
• traits::Tokenizer

Functions

• create_tokenizer_from_file
• create_tokenizer_from_file_with_options
• file_json_field
• hf::merge_special_tokens_from_config
• log_json_err

Type Aliases

• Offsets
• Result
• TokenIdType
• cache::CacheEventFn
• cache::CacheTokenUsageFn