Struct TokenizerState 

pub struct TokenizerState {
    pub vocab: HashMap<u32, String>,
    pub merges: Vec<(u32, u32, u32)>,
    pub special_tokens: HashMap<String, u32>,
}

A serializable snapshot of a trained tokenizer.

Fields

vocab: HashMap<u32, String>

id → token string

merges: Vec<(u32, u32, u32)>

(left_id, right_id, merged_id)

special_tokens: HashMap<String, u32>

special token name → id (e.g. "<BOS>" → 1)

Implementations

impl TokenizerState

pub fn new() -> Self

Create an empty state.

pub fn from_trained(trained: &TrainedTokenizer) -> Self

Build a TokenizerState from a crate::trainer::TrainedTokenizer.

pub fn vocab_size(&self) -> usize

Number of vocabulary entries.

pub fn save_to<W: Write>( &self, writer: &mut W, ) -> Result<(), SerializationError>

Save to a writer.

The format is deterministic: vocab entries are written sorted by id, merges in their original order, special tokens sorted by name.

pub fn load_from<R: BufRead>(reader: &mut R) -> Result<Self, SerializationError>

Load from a reader.

pub fn save(&self, path: &Path) -> Result<(), SerializationError>

Save to a file path.

pub fn load(path: &Path) -> Result<Self, SerializationError>

Load from a file path.

pub fn to_oxi_tokenizer(&self) -> OxiTokenizer

Convert to an crate::OxiTokenizer (char-level fallback using our vocab).

Trait Implementations

impl Debug for TokenizerState

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerState

fn default() -> Self

Returns the “default value” for a type.