Module cache 

Tokenizer caching layer (L1: prefix matching at special-token boundaries).

Wraps any Tokenizer in a cache that records prefix tokenizations at every special-token boundary. On a hit, the cached prefix tokens are merged with a fresh encode of the trailing suffix only — turning O(N) tokenization work into O(suffix_len) when prompts share a system prefix.

Correctness

Boundaries are taken only at positions immediately following a registered special token (e.g. <|im_start|>, <|im_end|>, <s>, </s>). Special tokens are atomic in BPE (special: true, normalized: false), so splitting there preserves the invariant tokenize(prefix) + tokenize(suffix) == tokenize(prefix + suffix). No fallback to whitespace or punctuation — better to miss than to corrupt.

Storage normalization

When L1 is enabled, every encode returns Encoding::Sp (token-ids only) — hits merge cached prefix ids with a fresh suffix encode, and misses assemble the ids from the per-boundary segment encodes (see L1Cache::populate_and_encode) — even when the inner tokenizer would have produced Encoding::Hf (rich offsets/attention/ etc). All current downstream consumers in Dynamo only call Encoding::token_ids, so this lossy normalization is safe; revisit if a caller starts reading offsets or attention masks from encodings produced through the cache.

Configuration

• special_tokens: Vec<String> — must be supplied at construction (the Tokenizer trait is intentionally minimal and does not expose them). An empty list disables L1: encode/encode_batch short-circuit straight to the inner tokenizer with no lookup, no miss-counter bump, and no insert attempt.
• max_memory_bytes — L1 byte budget; entries evicted via approximate LRU.

Provenance

Adapted from llm-tokenizer v1.3.2 (cache/l1.rs, cache/mod.rs). L0 and fingerprinting were dropped; L1 alone covers the headline multi-turn-chat workload, and the in-memory cache lifetime is bound to a single tokenizer instance so fingerprint-based invalidation is unnecessary.

Structs

CachedTokenizer

Caching wrapper around an inner tokenizer.

L1Cache

L1 cache: prefix matching at special-token boundaries, backed by a weighted W-TinyLFU moka cache that owns storage, recency/frequency tracking, and eviction. Hit/miss counts (our notion of a prefix hit) are tracked separately for metrics.

L1CacheStats

Type Aliases

CacheEventFn

Optional per-event observer. on_hit runs after each cache hit, on_miss after each miss — wired by CachedTokenizer::with_observer to push events straight into Prometheus counters without a periodic sampling step.