Struct LatentCache 

pub struct LatentCache { /* private fields */ }

Per-forward-pass, N-slot latent cache. See module docs for the shape contract and lifecycle.

Implementations

impl LatentCache

pub fn new( anchor: &Tensor, n_layers: usize, max_seq_len: usize, slot_trailing: Vec<Vec<usize>>, dtype: DType, ) -> Result<Self, Error>

Allocate a zero-filled cache on the same graph as anchor.

slot_trailing gives the trailing shape (past the seq axis) of each slot; its length is the number of slots per layer and must be ≥ 1. A slot’s buffer is [max_seq_len, …trailing] (an empty trailing ⇒ a [max_seq_len] per-token-scalar slot). dtype selects the element type (typically the latent’s — F32 / BF16 / F16 for inference).

pub fn append(self, layer: usize, new_slots: &[&Tensor]) -> Result<Self>

Append fresh latents for layer at the cache’s current position. Consumes self and returns the updated cache (option (b), the same functional shape as LazyKvCache::append).

new_slots must have exactly n_slots entries, in slot order; entry s must be [seqlen_new, …slot_trailing[s]], and all entries must agree on seqlen_new (the number of tokens appended this step: 1 for a decode step, prefill_len for prefill). Advances the cache’s position implicitly is not done here — call Self::advance_by after the last layer’s append in each step, matching LazyKvCache.

Errors (typed, at build time): layer out of bounds, wrong slot count, a slot shape mismatch, mismatched seqlen_new across slots, or an append that would exceed max_seq_len.

pub fn advance_by(self, n: usize) -> Self

Advance current_seq_len by n. Call after the last layer’s Self::append in each generation step (mirrors LazyKvCache::advance_by).

pub fn slot(&self, layer: usize, slot: usize) -> Tensor

Active slice of layer’s slot s: [0..current_seq_len] on dim 0 (clamped to ≥ 1 so a fresh cache still yields a valid rank).

pub fn slot_buffer_full(&self, layer: usize, slot: usize) -> Tensor

Full-capacity buffer for layer’s slot s ([max_seq, …trailing]) — escape hatch mirroring LazyKvCache::k_buffer_full.

pub fn current_seq_len(&self) -> usize

Tokens written so far.

pub fn max_seq_len(&self) -> usize

Pre-allocated per-slot capacity along the sequence axis.

pub fn n_layers(&self) -> usize

Number of layers.

pub fn n_slots(&self) -> usize

Number of slots per layer.

pub fn slot_trailing(&self, slot: usize) -> &[usize]

Trailing shape of slot s (past the seq axis).

Trait Implementations

impl Clone for LatentCache

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LatentCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.