Struct CachedTokenizer 

pub struct CachedTokenizer { /* private fields */ }

Caching wrapper around an inner tokenizer.

Implements Encoder, Decoder, and Tokenizer; decode calls pass through to the inner tokenizer (decoding is fast and rarely repeated).

Implementations

impl CachedTokenizer

pub fn new( inner: Arc<dyn Tokenizer>, special_tokens: Vec<String>, max_memory_bytes: usize, ) -> Result<Self>

Construct a cached tokenizer.

special_tokens is the list of atomic special-token strings the inner tokenizer recognizes (typically extracted via the HuggingFace tokenizer’s get_added_tokens_decoder() filtering by special == true). An empty list disables L1 — encode/encode_batch short-circuit to the inner tokenizer without touching the cache or its counters. An overlapping token set also disables L1, with a warning, because its boundaries are ambiguous.

max_memory_bytes is the L1 cache byte budget.

Errors

Returns the inner tokenizer’s compatibility error when it cannot be safely wrapped in the prefix cache.

pub fn with_extend(self, enabled: bool) -> Self

Enable partial-hit extension. When on, a partial cache hit also caches the freshly-tokenized suffix at its deepest special-token boundary, so each turn of a growing multi-turn conversation hits deeper than the last and per-turn tokenization cost stops growing with conversation length. Default off.

pub fn with_observer(self, on_hit: CacheEventFn, on_miss: CacheEventFn) -> Self

Install hit/miss callbacks so each L1 lookup pushes an event into the supplied closures (e.g. Prometheus::Counter::inc). Replaces any previously-set observer.

pub fn with_token_observer(self, observer: CacheTokenUsageFn) -> Self

Install a callback that receives exact cached and uncached token counts after each successful encode while L1 is active. A partial hit reports both categories, which lets consumers maintain token-level cache totals and derive a reuse ratio. Replaces any previously-set token observer.

This observer is not called when the special-token set is empty (and L1 is therefore disabled) or when encoding returns an error.

pub fn cache_stats(&self) -> L1CacheStats

Snapshot of L1 cache statistics (cumulative hits/misses/entries/memory).

pub fn clear_cache(&self)

Clear all cached entries and reset counters.

pub fn inner(&self) -> &Arc<dyn Tokenizer>

Access the underlying tokenizer (e.g. for downcasting to a concrete type).

Trait Implementations

impl Decoder for CachedTokenizer

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<DecodeResult>

impl Encoder for CachedTokenizer

fn encode(&self, input: &str) -> Result<Encoding>

fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>

fn encode_segments(&self, segments: &[EncodeSegment<'_>]) -> Result<Encoding>

Encode Kimi K3-style renderer segments while preserving trusted control-token and untrusted content boundaries.

impl Tokenizer for CachedTokenizer

fn vocab_size(&self) -> Option<usize>

Vocabulary cardinality including added tokens, when the backend can expose one. None for backends without a bounded id space or vocabulary introspection.

fn token_to_id(&self, token: &str) -> Result<Option<TokenIdType>>

Resolve a token string to its vocabulary id, when the backend supports lookup. Ok(None) when the token is not in the vocabulary. Err when this backend cannot do id lookup at all — kept distinct from a genuine vocabulary miss so callers can tell “unsupported” from “looked up, not found”.

fn special_token_ids(&self) -> Result<Vec<TokenIdType>>

Ids of added tokens marked special (e.g. BOS/EOS/PAD/control tokens), as distinct from ordinary vocabulary tokens. Ok(vec![]) for backends that genuinely have none, or that do not distinguish special from ordinary vocabulary ids. Err when the backend cannot enumerate added tokens at all — an empty Vec alone cannot carry that distinction.

fn num_special_tokens_added(&self) -> Result<usize>

Count of special tokens encode’s add_special_tokens: true path would add to a bare encoding (e.g. BOS/EOS), available without performing an encode. Ok(0) for backends that genuinely add none. Err when the backend cannot determine this at all — a plain 0 alone cannot carry that distinction, and this value feeds token-budget accounting where a silent 0 would under-count rather than fail loudly.

fn validate_prefix_cache(&self) -> Result<()>

Validate that this tokenizer can be safely wrapped in the prefix cache.

fn with_options(self, options: TokenizerOptions) -> Self

where Self: Sized,

Apply construction-time TokenizerOptions.