Struct CachedTokenizer 

pub struct CachedTokenizer { /* private fields */ }

Caching wrapper around an inner tokenizer.

Implements Encoder, Decoder, and Tokenizer; decode calls pass through to the inner tokenizer (decoding is fast and rarely repeated).

Implementations

impl CachedTokenizer

pub fn new( inner: Arc<dyn Tokenizer>, special_tokens: Vec<String>, max_memory_bytes: usize, ) -> Result<Self>

Construct a cached tokenizer.

special_tokens is the list of atomic special-token strings the inner tokenizer recognizes (typically extracted via the HuggingFace tokenizer’s get_added_tokens_decoder() filtering by special == true). An empty list disables L1 — encode/encode_batch short-circuit to the inner tokenizer without touching the cache or its counters. An overlapping token set also disables L1, with a warning, because its boundaries are ambiguous.

max_memory_bytes is the private token-ID payload byte budget. Moka defers eviction, so the budget is approximate and excludes keys and metadata.

Errors

Returns the inner tokenizer’s compatibility error when it cannot be safely wrapped in the prefix cache.

pub fn new_with_cache( inner: Arc<dyn Tokenizer>, special_tokens: Vec<String>, shared_cache: SharedTokenizerCache, namespace: &[u8], ) -> Result<Self>

Construct a tokenizer using shared storage and a caller-supplied namespace.

Equal namespaces share entries and must describe identical tokenizer behavior, including tokenizer files, backend, and encoding options. Different namespaces compete for the same byte budget but cannot reuse each other’s token IDs. Entries survive this wrapper being dropped while the shared cache remains alive. The eligibility checks are the same as Self::new.

Errors

Returns the inner tokenizer’s compatibility error if prefix caching is unsafe.

pub fn with_extend(self, enabled: bool) -> Self

Enable partial-hit extension. When on, a partial cache hit also caches the freshly-tokenized suffix at its deepest special-token boundary, so each turn of a growing multi-turn conversation hits deeper than the last and per-turn tokenization cost stops growing with conversation length. Default off.

pub fn with_observer(self, on_hit: CacheEventFn, on_miss: CacheEventFn) -> Self

Install hit/miss callbacks so each L1 lookup pushes an event into the supplied closures (e.g. Prometheus::Counter::inc). Replaces any previously-set observer.

pub fn with_token_observer(self, observer: CacheTokenUsageFn) -> Self

Install a callback that receives exact cached and uncached token counts after each successful encode while L1 is active. A partial hit reports both categories, which lets consumers maintain token-level cache totals and derive a reuse ratio. Replaces any previously-set token observer.

This observer is not called when the special-token set is empty (and L1 is therefore disabled) or when encoding returns an error.

pub fn cache_stats(&self) -> L1CacheStats

Wrapper-local hits/misses and namespace-wide entries/token bytes. Shared storage statistics scan the namespace; private caches use Moka’s totals. Results can change under concurrent writes. Disabled wrappers report zeroes.

pub fn inner(&self) -> &Arc<dyn Tokenizer>

Access the underlying tokenizer (e.g. for downcasting to a concrete type).

Trait Implementations

impl Decoder for CachedTokenizer

fn has_unstable_suffix( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> bool

Whether appending tokens can still reinterpret the decoded suffix. Callers must buffer an unstable suffix until a boundary or end of input.

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<DecodeResult>

impl Encoder for CachedTokenizer

fn encode(&self, input: &str) -> Result<Encoding>

fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>

fn encode_segments(&self, segments: &[EncodeSegment<'_>]) -> Result<Encoding>

Encode Kimi K3-style renderer segments while preserving trusted control-token and untrusted content boundaries.

impl Tokenizer for CachedTokenizer

fn vocab_size(&self) -> Option<usize>

Vocabulary cardinality including added tokens, when the backend can expose one. None for backends without a bounded id space or vocabulary introspection.

fn token_to_id(&self, token: &str) -> Result<Option<TokenIdType>>

Resolve a token string to its vocabulary id, when the backend supports lookup. Ok(None) when the token is not in the vocabulary. Err when this backend cannot do id lookup at all — kept distinct from a genuine vocabulary miss so callers can tell “unsupported” from “looked up, not found”.

fn special_token_ids(&self) -> Result<Vec<TokenIdType>>

Ids of added tokens marked special (e.g. BOS/EOS/PAD/control tokens), as distinct from ordinary vocabulary tokens. Ok(vec![]) for backends that genuinely have none, or that do not distinguish special from ordinary vocabulary ids. Err when the backend cannot enumerate added tokens at all — an empty Vec alone cannot carry that distinction.

fn num_special_tokens_added(&self) -> Result<usize>

Count of special tokens encode’s add_special_tokens: true path would add to a bare encoding (e.g. BOS/EOS), available without performing an encode. Ok(0) for backends that genuinely add none. Err when the backend cannot determine this at all — a plain 0 alone cannot carry that distinction, and this value feeds token-budget accounting where a silent 0 would under-count rather than fail loudly.

fn validate_prefix_cache(&self) -> Result<()>

Validate that this tokenizer can be safely wrapped in the prefix cache.

fn with_options(self, options: TokenizerOptions) -> Self

where Self: Sized,

Apply construction-time TokenizerOptions.