Enum AnyTokenizer 

pub enum AnyTokenizer {
    Bpe(Box<Gpt2Tokenizer>),
    Char(CharTokenizer),
}

Either tokenizer, for callers that pick one at runtime (the TUI and the wasm facade both do). Generation stays generic over impl Tokenizer, so this dispatches without a vtable.

Variants

Bpe(Box<Gpt2Tokenizer>)

Boxed because a Gpt2Tokenizer (four hash maps plus a compiled regex) is ~6× the size of a CharTokenizer, and every value of this enum would otherwise pay for the larger one.

Char(CharTokenizer)

Implementations

impl AnyTokenizer

pub fn bpe(t: Gpt2Tokenizer) -> Self

pub fn from_dir(dir: impl AsRef<Path>) -> Result<Self>

Load whichever tokenizer dir holds: BPE when merges.txt is present alongside vocab.json, otherwise the character vocab.

pub fn kind(&self) -> &'static str

Trait Implementations

impl Tokenizer for AnyTokenizer

fn encode(&self, text: &str) -> Result<Vec<u32>>

fn decode(&self, ids: &[u32]) -> String

fn decode_bytes(&self, ids: &[u32]) -> Vec<u8>

Raw byte-level decode. Must be append-only per token — i.e. decode_bytes(a ++ b) equals decode_bytes(a) ++ decode_bytes(b) — because the streaming path emits only the valid-UTF-8 prefix of the accumulated bytes.

fn vocab_size(&self) -> usize