Trait Tokenizer 

pub trait Tokenizer {
    // Required methods
    fn encode(&self, text: &str) -> Result<Vec<u32>>;
    fn decode(&self, ids: &[u32]) -> String;
    fn decode_bytes(&self, ids: &[u32]) -> Vec<u8> ;
    fn vocab_size(&self) -> usize;
}

The tokenizer surface the generation path needs — exactly the four methods crate::Gpt2::generate and friends call.

Required Methods

fn encode(&self, text: &str) -> Result<Vec<u32>>

fn decode(&self, ids: &[u32]) -> String

fn decode_bytes(&self, ids: &[u32]) -> Vec<u8>

Raw byte-level decode. Must be append-only per token — i.e. decode_bytes(a ++ b) equals decode_bytes(a) ++ decode_bytes(b) — because the streaming path emits only the valid-UTF-8 prefix of the accumulated bytes.

fn vocab_size(&self) -> usize

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl Tokenizer for AnyTokenizer

impl Tokenizer for CharTokenizer

impl Tokenizer for Gpt2Tokenizer

Delegates to the inherent methods, which stay public so existing callers compile unchanged (inherent methods win over trait methods at the call site, so this is not recursive).