Struct LocalTokenizer 

pub struct LocalTokenizer { /* private fields */ }

A tokenizer loaded from a local tokenizer.json.

Implementations

impl LocalTokenizer

pub fn from_file(path: &Path) -> Result<Self>

Load a tokenizer.json from disk.

pub fn from_bytes(bytes: &[u8]) -> Result<Self>

Load a tokenizer.json already in memory.

pub fn tokenize(&self, content: &str) -> Result<Vec<u32>>

Encode without special tokens, matching fastjev’s encode_prompt.

pub fn detokenize(&self, tokens: &[u32]) -> Result<String>

Decode skipping special tokens, matching fastjev’s round-trip check.

pub fn vocab_size(&self) -> usize

The vocabulary size, for startup diagnostics.

pub fn verify_special_tokens(&self, expected: &Map<String, Value>) -> Result<()>

Refuse a tokenizer whose special tokens disagree with the template.

A template that emits {{ bos_token }} and a tokenizer whose BOS token is something else would produce a prompt the model never saw in training, so this is checked rather than assumed.