Serializable

smoltok_core::tokenizer

Trait Serializable 

Source 
pub trait Serializable: Tokenizer {
    // Required method
    fn save(&self, path: &Path) -> Result<(), Error>;
}
Expand description

A trait for serializing a tokenizer to a file.

This trait provides a default implementation for saving tokenizer merge rules to a .stok file format.

Required Methods§

Source

fn save(&self, path: &Path) -> Result<(), Error>

Saves the tokenizer’s merge rules to a .stok file and vocabulary to a .vocab file.

§Arguments
  • path - The path to save the tokenizer. Must have a .stok extension.
§Returns
  • Ok(()) if the tokenizer was saved successfully.
  • Err(std::io::Error) if the file extension is invalid or writing fails.

Implementors§