Skip to main contentCrate autotokenizer
Source - AutoTokenizer
- Mirrors the fields of
tokenizer_config.json required for chat-template
rendering and basic token identity. Unknown JSON keys are silently ignored. - ChatTemplateEntry
- One entry in a multi-template
chat_template list. - DefaultPromptMessage
- A single conversation turn for use with
AutoTokenizer::apply_chat_template. - FromPretrainedParameters
- Defines the aditional parameters available for the
from_pretrained function - TokenObj
- The expanded form of a special token as stored in
tokenizer_config.json.
- ChatTemplate
chat_template in tokenizer_config.json is either a plain Jinja string
or a list of {"name": …, "template": …} objects (e.g. for tool-use models).- Token
- A special token that can appear either as a plain string or as an object.