Struct CachedTokenizer 

pub struct CachedTokenizer { /* private fields */ }

Caching wrapper around an inner tokenizer.

Implements Encoder, Decoder, and Tokenizer; decode calls pass through to the inner tokenizer (decoding is fast and rarely repeated).

Implementations

impl CachedTokenizer

pub fn new( inner: Arc<dyn Tokenizer>, special_tokens: Vec<String>, max_memory_bytes: usize, ) -> Self

Construct a cached tokenizer.

special_tokens is the list of atomic special-token strings the inner tokenizer recognizes (typically extracted via the HuggingFace tokenizer’s get_added_tokens_decoder() filtering by special == true). An empty list disables L1 — encode/encode_batch short-circuit to the inner tokenizer without touching the cache or its counters.

max_memory_bytes is the L1 cache byte budget.

pub fn with_extend(self, enabled: bool) -> Self

Enable partial-hit extension. When on, a partial cache hit also caches the freshly-tokenized suffix at its deepest special-token boundary, so each turn of a growing multi-turn conversation hits deeper than the last and per-turn tokenization cost stops growing with conversation length. Default off.

pub fn with_observer(self, on_hit: CacheEventFn, on_miss: CacheEventFn) -> Self

Install hit/miss callbacks so each L1 lookup pushes an event into the supplied closures (e.g. Prometheus::Counter::inc). Replaces any previously-set observer.

pub fn with_token_observer(self, observer: CacheTokenUsageFn) -> Self

Install a callback that receives exact cached and uncached token counts after each successful encode while L1 is active. A partial hit reports both categories, which lets consumers maintain token-level cache totals and derive a reuse ratio. Replaces any previously-set token observer.

This observer is not called when the special-token set is empty (and L1 is therefore disabled) or when encoding returns an error.

pub fn cache_stats(&self) -> L1CacheStats

Snapshot of L1 cache statistics (cumulative hits/misses/entries/memory).

pub fn clear_cache(&self)

Clear all cached entries and reset counters.

pub fn inner(&self) -> &Arc<dyn Tokenizer>

Access the underlying tokenizer (e.g. for downcasting to a concrete type).

Trait Implementations

impl Decoder for CachedTokenizer

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<DecodeResult>

impl Encoder for CachedTokenizer

fn encode(&self, input: &str) -> Result<Encoding>

fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>

impl Tokenizer for CachedTokenizer

fn with_options(self, options: TokenizerOptions) -> Self

where Self: Sized,

Apply construction-time TokenizerOptions.