Struct TiktokenTokenizer 

pub struct TiktokenTokenizer { /* private fields */ }

Tiktoken tokenizer wrapper — supports both built-in OpenAI encodings and hub-loaded models.

Implementations

impl TiktokenTokenizer

pub fn new(model: TiktokenModel) -> Result<Self>

Create a new Tiktoken tokenizer for the specified built-in model

pub fn from_dir(dir: &Path) -> Result<Self>

Create from a directory containing tiktoken.model + tokenizer_config.json

pub fn from_dir_with_chat_template( dir: &Path, chat_template_path: Option<&str>, ) -> Result<Self>

Create from a directory with an optional chat template file path. Discovers the tiktoken model file automatically via find_tiktoken_file.

pub fn from_file(tiktoken_path: &Path) -> Result<Self>

Create from an exact tiktoken file path (.tiktoken or tiktoken.model). Looks for tokenizer_config.json in the same directory.

pub fn from_file_with_chat_template( tiktoken_path: &Path, chat_template_path: Option<&str>, ) -> Result<Self>

Create from an exact tiktoken file path with an optional chat template.

pub fn from_model_name(model_name: &str) -> Result<Self>

Create a tokenizer from a model string (e.g., “gpt-4”, “gpt-3.5-turbo”)

Trait Implementations

impl Decoder for TiktokenTokenizer

fn decode( &self, token_ids: &[TokenIdType], _skip_special_tokens: bool, ) -> Result<String>

fn decode_step( &self, token_id: TokenIdType, ids: &mut Vec<TokenIdType>, prefix: &mut String, prefix_index: &mut usize, skip_special_tokens: bool, ) -> Result<Option<String>>

Incremental decode step — called once per generated token.

impl Encoder for TiktokenTokenizer

fn encode(&self, input: &str, _add_special_tokens: bool) -> Result<Encoding>

fn encode_batch( &self, inputs: &[&str], add_special_tokens: bool, ) -> Result<Vec<Encoding>>

impl Tokenizer for TiktokenTokenizer

fn vocab_size(&self) -> usize

fn get_special_tokens(&self) -> &SpecialTokens

fn token_to_id(&self, token: &str) -> Option<TokenIdType>

fn id_to_token(&self, id: TokenIdType) -> Option<String>

fn as_any(&self) -> &dyn Any

Enable downcasting to concrete types

fn apply_chat_template( &self, messages: &[Value], params: ChatTemplateParams<'_>, ) -> Result<String>

Apply chat template to messages. Default returns an error for tokenizers without template support.

fn chat_template_content_format(&self) -> ChatTemplateContentFormat

Get the content format expected by the chat template.

fn thinking_toggle(&self) -> ThinkingToggle

Get the thinking toggle support for this template.

fn thinking_key_name(&self) -> Option<ThinkingKeyName>

The variable name the template uses for the thinking toggle.

fn think_in_prefill(&self) -> bool

Whether the template injects <think> in the generation prompt.

fn set_chat_template(&mut self, template: String) -> Result<()>

Set or override the chat template.