Struct TokenizerSpec 

pub struct TokenizerSpec {
    pub tokenizer_json: PathBuf,
    pub added_tokens: HashMap<u32, String>,
    pub chat_template: Option<String>,
    pub add_bos: Option<bool>,
}

Where to find the tokenizer and which special tokens it defines.

Fields

tokenizer_json: PathBuf

Path to the HuggingFace tokenizer.json.

added_tokens: HashMap<u32, String>

Added special tokens parsed from tokenizer_config.json (added_tokens_decoder): token id → token string (e.g. <|im_end|>).

chat_template: Option<String>

Raw chat template string (Jinja) if present in tokenizer_config.json. Phase 1 uses a built-in ChatML wrap instead of evaluating Jinja.

add_bos: Option<bool>

Whether prompts should be prefixed with BOS (HF add_bos_token / GGUF tokenizer.ggml.add_bos_token). None = unspecified, which the engine treats as “prepend when the model declares a BOS id”. Qwen2 declares a BOS id but sets this to false; ignoring it prepends <|endoftext|> to every prompt.

Implementations

impl TokenizerSpec

pub fn special_token_id(&self, token: &str) -> Option<u32>

Looks up the id of an added special token by its string, e.g. spec.special_token_id("<|im_end|>").

pub fn chatml_wrap(&self, user_prompt: &str) -> Option<String>

Wraps a user prompt in the ChatML template used by SmolLM2-style instruction models. Returns None if the tokenizer does not define <|im_start|> / <|im_end|> tokens.

pub fn chat_template_kind(&self) -> ChatTemplate

Which chat template this tokenizer’s special tokens imply.

pub fn wrap_messages(&self, messages: &[(String, String)]) -> String

Wraps (role, content) message pairs into the model’s chat format, ending with the assistant turn left open for generation. Unknown roles are coerced to user (same convention as serve).

Trait Implementations

impl Clone for TokenizerSpec

fn clone(&self) -> TokenizerSpec

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerSpec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.