Struct TiktokenTokenizer 

pub struct TiktokenTokenizer { /* private fields */ }

OpenAI tiktoken-based tokenizer.

Supports cl100k_base (GPT-4) and o200k_base (GPT-4o) encodings.

Implementations

impl TiktokenTokenizer

pub fn cl100k() -> Result<Self>

Create cl100k_base tokenizer (GPT-3.5, GPT-4).

Errors

Returns error if tokenizer initialization fails.

pub fn o200k() -> Result<Self>

Create o200k_base tokenizer (GPT-4o, o1, o3).

Errors

Returns error if tokenizer initialization fails.

pub fn from_type(tokenizer_type: TokenizerType) -> Result<Self>

Create tokenizer from type.

Errors

Returns error if tokenizer initialization fails or type is not tiktoken-based.

Trait Implementations

impl HydraTokenizer for TiktokenTokenizer

fn encode(&self, text: &str) -> Result<Vec<u32>>

Encode text to token IDs.

fn decode(&self, tokens: &[u32]) -> Result<String>

Decode token IDs back to text.

fn vocab_size(&self) -> usize

Get the vocabulary size.

fn tokenizer_type(&self) -> TokenizerType

Get the tokenizer type.

fn truncate(&self, tokens: Vec<u32>) -> Vec<u32>

Truncate tokens to maximum length for Hydra.

fn encode_for_hydra(&self, text: &str) -> Result<Vec<u32>>

Encode and truncate for Hydra input.