Struct TokenizerBridge 

pub struct TokenizerBridge { /* private fields */ }

Thin wrapper around tokenizers::Tokenizer.

On non-WASM targets, delegates to the full HuggingFace tokenizers library. On WASM targets, all methods return a RuntimeError::Tokenizer error.

Implementations

impl TokenizerBridge

pub fn from_file(path: &str) -> RuntimeResult<Self>

Load a tokenizer from a JSON file.

pub fn encode(&self, text: &str) -> RuntimeResult<Vec<u32>>

Encode text to token IDs.

pub fn decode(&self, ids: &[u32]) -> RuntimeResult<String>

Decode token IDs to text.

pub fn vocab_size(&self) -> usize

Get the vocabulary size.

pub fn inner(&self) -> &Tokenizer

Get the internal tokenizer reference.

pub fn new_decode_stream(&self, skip_special_tokens: bool) -> DecodeStreamState

Create a fresh decode-stream state for one generation request.

See DecodeStreamState and Self::step_decode for the streaming decode protocol. Use this instead of repeatedly calling Self::decode with single-token slices, which mishandles tokens that straddle UTF-8 codepoint boundaries.

pub fn step_decode( &self, state: &mut DecodeStreamState, id: u32, ) -> RuntimeResult<Option<String>>

Advance the decode stream by one token.

Returns Ok(Some(text)) only when the buffered bytes form a complete UTF-8 chunk (which may span several previous tokens for CJK / emoji); returns Ok(None) when more tokens are needed before any well-formed text can be emitted. Callers must not print the empty string when Ok(None) is returned — wait for the next token.