Struct TikTokenTokenizer 

pub struct TikTokenTokenizer { /* private fields */ }

Implementations

impl TikTokenTokenizer

pub fn from_file( path: &str, pattern: &str, special_tokens: FxHashMap<String, u32>, ) -> Result<Self>

Create a TikTokenTokenizer from a tiktoken model file.

Arguments

• path - Path to the .model or .tiktoken file (base64 rank-per-line format)
• pattern - BPE regex pattern string
• special_tokens - Map of special token strings to their IDs

pub fn from_file_auto(path: &str) -> Result<Self>

Create a TikTokenTokenizer from a tiktoken model file, auto-detecting the BPE pattern from config.json and special tokens from tokenizer_config.json.

The tiktoken file and config files must be in the same directory.

pub fn special_tokens(&self) -> &[String]

Atomic special-token strings registered with the underlying TikToken BPE.

CoreBPE::encode_with_special_tokens recognizes these strings outside ordinary BPE. For non-overlapping token spellings, splitting immediately after a match preserves tokenization.

Trait Implementations

impl Decoder for TikTokenTokenizer

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<DecodeResult>

impl Encoder for TikTokenTokenizer

fn encode(&self, input: &str) -> Result<Encoding>

fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>

impl Tokenizer for TikTokenTokenizer

fn with_options(self, options: TokenizerOptions) -> Self

where Self: Sized,

Apply construction-time TokenizerOptions.