Struct Tokenizer 

pub struct Tokenizer {
    pub vocab: HashMap<String, u32>,
    pub id_to_token: HashMap<u32, String>,
    pub bos_token_id: u32,
    pub eos_token_id: u32,
    pub pad_token_id: u32,
    pub hf_tokenizer: Option<Arc<Tokenizer>>,
}

Wraps a Hugging Face tokenizer.json with special-token ids.

Fields

vocab: HashMap<String, u32>

Token string to id.

id_to_token: HashMap<u32, String>

Id to token string.

bos_token_id: u32

Beginning-of-sequence token id.

eos_token_id: u32

End-of-sequence token id.

pad_token_id: u32

Padding token id.

hf_tokenizer: Option<Arc<Tokenizer>>

Underlying Hugging Face tokenizer, if it loaded.

Implementations

impl Tokenizer

pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self, String>

Load tokenizer from tokenizer.json

pub fn encode(&self, text: &str) -> Vec<u32>

Encode text to token IDs, adding the tokenizer’s special tokens (e.g. a leading BOS) when it defines any.

pub fn encode_continuation(&self, text: &str) -> Vec<u32>

Encode text without adding special tokens; use this for text that continues a sequence (an SFT response after its prompt).

pub fn decode(&self, tokens: &[u32]) -> String

Decode token IDs back to text

pub fn chat_prompt(system: Option<&str>, user: &str) -> String

The ChatML prompt for one user turn (with an optional system turn), ending with the assistant header so the model continues as the assistant: <|im_start|>user\n{user}<|im_end|>\n<|im_start|>assistant\n.

pub fn encode_chat(&self, system: Option<&str>, user: &str) -> Vec<u32>

Encode chat_prompt.

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.