Struct TokenizerSpec 

pub struct TokenizerSpec {
    pub tokenizer_json: PathBuf,
    pub added_tokens: HashMap<u32, String>,
    pub chat_template: Option<String>,
}

Where to find the tokenizer and which special tokens it defines.

Fields

tokenizer_json: PathBuf

Path to the HuggingFace tokenizer.json.

added_tokens: HashMap<u32, String>

Added special tokens parsed from tokenizer_config.json (added_tokens_decoder): token id → token string (e.g. <|im_end|>).

chat_template: Option<String>

Raw chat template string (Jinja) if present in tokenizer_config.json. Phase 1 uses a built-in ChatML wrap instead of evaluating Jinja.

Implementations

impl TokenizerSpec

pub fn special_token_id(&self, token: &str) -> Option<u32>

Looks up the id of an added special token by its string, e.g. spec.special_token_id("<|im_end|>").

pub fn chatml_wrap(&self, user_prompt: &str) -> Option<String>

Wraps a user prompt in the ChatML template used by SmolLM2-style instruction models. Returns None if the tokenizer does not define <|im_start|> / <|im_end|> tokens.

Trait Implementations

impl Clone for TokenizerSpec

fn clone(&self) -> TokenizerSpec

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerSpec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.