Struct V3Cache 

pub struct V3Cache { /* private fields */ }

Decode state for one session.

Implementations

impl V3Cache

pub fn new(cfg: &V3Config, hint: usize) -> Self

A cache for a session expected to reach hint positions.

hint only sizes the first allocation for global layers — exceeding it grows rather than fails. Local layers are always exactly their window.

pub fn with_precision( cfg: &V3Config, hint: usize, precision: KvPrecision, ) -> Self

A cache that stores its keys and values at precision from the first write.

Precision is fixed at construction because it decides the allocation: an Int8 cache never allocates the f32 buffers at all, which is the entire point. Switching afterwards would either reinterpret what is already stored or silently throw it away.

pub fn precision(&self) -> KvPrecision

pub fn len(&self) -> usize

Positions written so far.

pub fn is_empty(&self) -> bool

pub fn bytes(&self) -> usize

Bytes currently held, so a caller on a constrained target can see what a session actually costs rather than guessing.

pub fn token_history(&self) -> &[u32]

Token ids retained for the n-gram hash, oldest first.

pub fn push_token(&mut self, tok: u32)

Record a token, dropping history no longer reachable by any hash.

pub fn span(&self, li: usize, pos: usize) -> (usize, usize)

The inclusive logical range layer li may attend to at pos.

Clamped to the ring’s capacity. A global layer past max_seq_len would otherwise ask for more history than it stores, and because slots are addressed modulo the capacity it would read itself as its own past — silently, since the bounds check is a debug assertion. Degrading to the most recent slots positions is wrong too, but it is bounded, visible through Self::is_saturated, and cannot corrupt.

Callers should avoid reaching this: see V3Engine, which bounds the prompt and reports StopReason::MaxSeqLen.

pub fn is_saturated(&self) -> bool

True once any layer has run past what it can store, so attention is no longer seeing the history the model was trained to see.

pub fn slots(&self, li: usize) -> usize

Ring capacity for layer li.

pub fn write_kv(&mut self, li: usize, pos: usize, k: &[f32], v: &[f32])

Write this position’s post-convolution key and value.

pub fn kv(&self, li: usize) -> KvStore<'_>

How layer li hands its keys and values to attention.

pub fn conv_step( &mut self, li: usize, which: Qkv, taps: &[f32], raw: &mut [f32], )

Apply the causal conv for one position using the retained raw tail, then roll the tail forward.

raw is this position’s pre-conv projection and is replaced with the post-conv result. History before the start of the session contributes zero, exactly as the reference’s zero-padded shift does.

pub fn engram_conv_step(&mut self, site: usize, taps: &[f32], raw: &mut [f32])

The Engram value convolution for one position, using the retained tail.

raw is this position’s pre-conv Engram value and is replaced with the post-conv result.

pub fn seed_conv_tail(&mut self, li: usize, which: Qkv, raw: &[f32])

Seed the conv tail for a layer after a batched prefill.

raw holds the last qkv_conv_taps - 1 pre-convolution vectors for this projection, oldest first. The batched path convolves in place, so the caller has to keep these before it does.

pub fn seed_engram_tail(&mut self, site: usize, raw: &[f32])

Seed one site’s Engram value tail, oldest first, pre-convolution.

pub fn seed_tokens(&mut self, tokens: &[u32])

Seed the token history the n-gram hash reads, oldest first.

pub fn set_pos(&mut self, pos: usize)

Set the next write position after a batched prefill.

pub fn advance(&mut self)

Advance to the next position. Call once per decoded token, after every layer has written its key and value.

pub fn pos(&self) -> usize

Current write position.