Struct Gpt2Tokenizer 

pub struct Gpt2Tokenizer { /* private fields */ }

Implementations

impl Gpt2Tokenizer

pub fn from_files( vocab_path: impl AsRef<Path>, merges_path: impl AsRef<Path>, ) -> Result<Self>

pub fn from_strs(vocab_json: &str, merges: &str) -> Result<Self>

Build from the contents of vocab.json and merges.txt — the primary form; on wasm the strings arrive from an HTTP fetch.

pub fn from_dir(dir: impl AsRef<Path>) -> Result<Self>

Load vocab.json + merges.txt from a directory.

pub fn encode(&self, text: &str) -> Result<Vec<u32>>

pub fn decode(&self, ids: &[u32]) -> String

pub fn decode_bytes(&self, ids: &[u32]) -> Vec<u8>

Raw byte-level decode. Unlike Gpt2Tokenizer::decode this is exact and append-only per token (id sequence a ++ b decodes to bytes(a) ++ bytes(b)), which streaming builds on: a multi-byte UTF-8 character split across BPE tokens completes once the next token’s bytes arrive.

pub fn vocab_size(&self) -> usize

Trait Implementations

impl Tokenizer for Gpt2Tokenizer

Delegates to the inherent methods, which stay public so existing callers compile unchanged (inherent methods win over trait methods at the call site, so this is not recursive).

fn encode(&self, text: &str) -> Result<Vec<u32>>

fn decode(&self, ids: &[u32]) -> String

fn decode_bytes(&self, ids: &[u32]) -> Vec<u8>

Raw byte-level decode. Must be append-only per token — i.e. decode_bytes(a ++ b) equals decode_bytes(a) ++ decode_bytes(b) — because the streaming path emits only the valid-UTF-8 prefix of the accumulated bytes.

fn vocab_size(&self) -> usize