Struct Tokenizer 

pub struct Tokenizer {
    pub bos_token_id: Option<u32>,
    pub eos_token_id: Option<u32>,
    pub pad_token_id: Option<u32>,
    pub im_start_id: Option<u32>,
    pub im_end_id: Option<u32>,
    pub chat_template: Option<String>,
    pub extra_eos: HashSet<u32>,
    pub add_bos: bool,
    /* private fields */
}

A loaded BPE tokenizer.

Fields

bos_token_id: Option<u32>

Special tokens

eos_token_id: Option<u32>

pad_token_id: Option<u32>

im_start_id: Option<u32>

Chat template special tokens

im_end_id: Option<u32>

chat_template: Option<String>

Jinja chat template carried by the container (spec §6.1); None → hardcoded ChatML fallback.

extra_eos: HashSet<u32>

Extra stop ids from the container’s generation config.

add_bos: bool

Generation prepends BOS (llama post_processor semantics).

Implementations

impl Tokenizer

pub fn from_file(path: impl AsRef<Path>) -> Result<Self, TokenizerError>

Load tokenizer from HuggingFace tokenizer.json file.

pub fn from_bytes(bytes: &[u8]) -> Result<Self, TokenizerError>

Load tokenizer from raw tokenizer.json bytes (CMF VOCAB section).

pub fn from_json(json: &str) -> Result<Self, TokenizerError>

Load tokenizer from JSON string.

pub fn byte_level() -> Self

Create a minimal tokenizer for testing (byte tokens, no merges).

pub fn encode(&self, text: &str) -> Vec<u32>

Encode text to token IDs.

pub fn decode(&self, ids: &[u32]) -> String

Decode token IDs back to text. Special tokens are skipped; added tokens are raw text; everything else reverses the byte-level map.

pub fn decode_token(&self, id: u32) -> String

Streaming decode of ONE token: no sequence-level Strip — a per-token strip would eat the ▁-spaces of every SP word.

pub fn apply_chat_template(&self, messages: &[(String, String)]) -> Vec<u32>

Render the container’s Jinja chat template (HF semantics: trim_blocks + lstrip_blocks + loop controls) and encode it. Falls back to hardcoded ChatML when the file carries none.

pub fn with_bos(&self, ids: Vec<u32>) -> Vec<u32>

Prepend BOS when the tokenizer declares it (llama family).

pub fn render_chat(&self, messages: &[(String, String)]) -> Option<String>

Render the carried template to text (parity-testable surface).

pub fn vocab_size(&self) -> usize

Vocabulary size.

pub fn is_eos(&self, id: u32) -> bool

Check if token ID is EOS.

Trait Implementations

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.