Struct TekkenEncoder 

pub struct TekkenEncoder { /* private fields */ }

Tekken BPE encoder for text → token IDs.

Wraps tiktoken’s CoreBPE with the correct vocab and regex pattern from a tekken.json file. Output token IDs include the 1000 offset (matching the model’s embedding table layout).

Implementations

impl TekkenEncoder

pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self>

Load encoder from a tekken.json file.

pub fn from_json(json: &str) -> Result<Self>

Load encoder from a JSON string.

pub fn encode(&self, text: &str) -> Vec<u32>

Encode text to token IDs (with 1000 offset applied).

Returns token IDs matching the model’s embedding table layout: IDs 0-999 are special tokens, 1000+ are text tokens.