Struct L1Cache 

pub struct L1Cache { /* private fields */ }

L1 cache: prefix matching at special-token boundaries, backed by a weighted W-TinyLFU moka cache that owns storage, recency/frequency tracking, and eviction. Hit/miss counts (our notion of a prefix hit) are tracked separately for metrics.

Implementations

impl L1Cache

pub fn new(max_memory: usize, special_tokens: Vec<String>) -> Self

special_tokens is the atomic special-token set whose boundaries the cache splits at; an empty set leaves L1 inert (no boundaries, no entries).

pub fn set_observer(&mut self, on_hit: CacheEventFn, on_miss: CacheEventFn)

Install hit/miss callbacks. Replaces any previously-set observers.

pub fn longest_prefix_match( &self, input: &str, ) -> Option<(Vec<TokenIdType>, usize, usize)>

Try to find the longest prefix match at a special-token boundary.

Returns (cached_tokens, byte_offset, deepest_boundary) if found. The caller extends the cached tokens with a fresh encode of input[byte_offset..]; deepest_boundary is the deepest special-token boundary in input (end-exclusive), handed back so [extend_after_match] need not rescan the input for it.

pub fn insert_at_boundaries<E: Encoder + ?Sized>( &self, input: &str, tokenizer: &E, ) -> Result<()>

Insert prefix entries at every special-token boundary (e.g. to pre-seed the cache).

Uses incremental hashing and incremental tokenization (per-segment encode of the delta text between adjacent boundaries) so populating N entries costs one full re-tokenize total, split across the segments. The miss path uses Self::populate_and_encode instead, which reuses this same work to also return the full token vector (avoiding a redundant second tokenization).

pub fn populate_and_encode<E: Encoder + ?Sized>( &self, input: &str, tokenizer: &E, ) -> Result<Vec<TokenIdType>>

Miss-path encode: tokenize input exactly once, caching the cumulative prefix at every special-token boundary as we go, and return the full token-id vector. This replaces a separate full encode + Self::insert_at_boundaries, which together tokenized the input ~twice (once for the result, once split across segments).

The concatenation of the per-segment encodes equals an uncached encode(input) because special tokens are atomic in BPE — the same invariant the hit path relies on. Returns token-ids only; the caller wraps them in crate::Encoding::Sp.

pub fn extend_after_match<E: Encoder + ?Sized>( &self, input: &str, prefix_tokens: Vec<TokenIdType>, prefix_len: usize, deepest_boundary: usize, tokenizer: &E, ) -> Result<Vec<TokenIdType>>

Extend the cache on a partial hit so the next turn of a growing conversation hits deeper. Given the (prefix_tokens, prefix_len, deepest_boundary) returned by [longest_prefix_match], tokenize the remaining suffix and cache the cumulative prefix at the suffix’s deepest special-token boundary, then return the full merged token vector.

Deepest-only is intentional: in an append-only conversation the next turn always reaches the deepest boundary, so caching it bounds per-turn work to the newest exchange; shallow/branching coverage already comes from the miss path’s [insert_at_boundaries]. Splitting at special-token boundaries is correctness-safe because special tokens are atomic in BPE (tokenize(a) + tokenize(b) == tokenize(a + b)).

Note: unlike the read-only fast path, this writes to the cache on a hit (one insert + possible eviction). It relies on the same best-effort memory accounting as [insert_at_boundaries].

pub fn len(&self) -> usize

Number of live entries. Flushes moka’s deferred maintenance first so the count is exact rather than lagging behind pending inserts/evictions.

pub fn is_empty(&self) -> bool

pub fn stats(&self) -> L1CacheStats

pub fn clear(&self)