Struct Llama3Tokenizer 

pub struct Llama3Tokenizer { /* private fields */ }

Llama 3 tokenizer using HuggingFace Tokenizers library.

This is the primary tokenizer for Hydra, supporting the 128K vocabulary used by Llama 3 and compatible models.

Example

let tokenizer = Llama3Tokenizer::from_file("./tokenizer.json")?;
let tokens = tokenizer.encode("Hello, world!")?;

Implementations

impl Llama3Tokenizer

pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self>

Load tokenizer from a tokenizer.json file.

Errors

Returns error if the file cannot be read or parsed.

pub fn from_json(json: &str) -> Result<Self>

Load tokenizer from JSON string.

Errors

Returns error if the JSON is invalid.

pub fn from_bytes(bytes: &[u8]) -> Result<Self>

Load tokenizer from bytes.

Errors

Returns error if the bytes are not valid JSON.

Trait Implementations

impl HydraTokenizer for Llama3Tokenizer

fn encode(&self, text: &str) -> Result<Vec<u32>>

Encode text to token IDs.

fn decode(&self, tokens: &[u32]) -> Result<String>

Decode token IDs back to text.

fn vocab_size(&self) -> usize

Get the vocabulary size.

fn tokenizer_type(&self) -> TokenizerType

Get the tokenizer type.

fn truncate(&self, tokens: Vec<u32>) -> Vec<u32>

Truncate tokens to maximum length for Hydra.

fn encode_for_hydra(&self, text: &str) -> Result<Vec<u32>>

Encode and truncate for Hydra input.