Struct LatentKvCache 

pub struct LatentKvCache {
    pub layers: Vec<Option<Vec<LatentSlot>>>,
    pub cached_len: usize,
    pub slot_trailing: Vec<Vec<usize>>,
    pub max_seq_len: usize,
    pub dtype: DType,
    /* private fields */
}

Cross-graph, cross-forward-pass persistent N-slot decode cache — the KvCache generalization for latent-caching attention architectures.

KvCache hardwires a symmetric K/V pair (two buffers per layer, same shape). That’s wrong for compression architectures:

• Multi-head Latent Attention (DeepSeek-V2 MLA): per layer, a low-rank compressed latent [kv_lora_rank] (slot 0) and a single-head RoPE key [qk_rope_head_dim] (slot 1) — two slots of different trailing shape, neither a [n_kv_heads, head_dim] K/V.
• Two-projection attention / QKV pruning: a single retained projection per layer — one slot.

LatentKvCache is this module’s counterpart to crate::lazy_latent_cache::LatentCache — that type makes the exact same per-layer, ordered list of latent buffers with independent trailing shapes generalization for the per-forward- pass (single fuel_graph::Graph-anchored, functional append-and-thread) lifecycle. LatentKvCache is the persistent sibling: device-resident Arc<RwLock<Storage>> buffers that survive across graphs/forward calls and are mutated in place via Op::WriteSlice, exactly the relationship KvCache has to a plain per-pass K/V cache. Read both modules’ docs together for the two lifecycles side by side.

Shape contract

Slot s of every layer is a buffer [max_seq_len, ...slot_trailing[s]] — the capacity axis is dim 0. This differs from KvCache’s [1, n_kv_heads, max_seq_len, head_dim] convention (capacity axis 2): a latent is a per-token vector, not a per-head plane, so there is no head axis to lead with. Matches [crate::lazy_latent_cache:: LatentCache]’s dim-0 convention exactly — this type’s per-slot buffer is byte-for-byte what that type’s slot_buffer_full realizes, modulo device residency.

Fields

layers: Vec<Option<Vec<LatentSlot>>>

layers[l] is None until populated; Some(slots) holds one LatentSlot per slot index (pipeline-parallel friendly — mirrors KvCache::layers).

cached_len: usize

slot_trailing: Vec<Vec<usize>>

Per-slot trailing shape (past the leading capacity axis); its length is the per-layer slot count.

max_seq_len: usize

dtype: DType

Implementations

impl LatentKvCache

pub fn with_capacity( n_layers: usize, max_seq_len: usize, slot_trailing: Vec<Vec<usize>>, dtype: DType, device: &Device, ) -> Result<Self>

Pre-allocated N-slot latent cache. Every layer’s slot buffers are allocated up-front as zero buffers on device with dtype, shaped [max_seq_len, ...slot_trailing[s]]. Mirrors [KvCache:: with_capacity]’s Op::Alloc → Op::ZeroFill graph-emission pattern exactly, generalized from a fixed 2-buffer (K, V) layer to slot_trailing.len() buffers of independent shape.

Returns Err for degenerate geometry (n_layers == 0, max_seq_len == 0, or no slots) or if any per-layer allocation fails (e.g. CUDA OOM / an unwired device).

pub fn slot_storage( &self, layer_idx: usize, slot: usize, ) -> Option<Arc<RwLock<Storage>>>

Borrow the slot s storage Arc for layer layer_idx. None if the layer is unpopulated or either index is out of range. Used by the forward path to bind cache storage to per-step Const nodes via InferenceContext::insert. Mirrors KvCache::slot_storage.

pub fn bump_version(&mut self, layer_idx: usize, slot: usize)

Bump slot s’s monotonic version counter for layer_idx. No-op (never panics) if either index is out of range. Mirrors KvCache::bump_version.

pub fn n_layers(&self) -> usize

pub fn n_slots(&self) -> usize

Number of slots per layer.

pub fn slot_trailing(&self, s: usize) -> &[usize]

Trailing shape of slot s (past the leading capacity axis).

pub fn alloc_id(&self) -> KvAllocId

Identity of the storage this cache currently owns. Mirrors KvCache::alloc_id; see KvAllocId.

pub fn clear(&mut self)

Drop every layer; reset cached_len to zero. Mirrors KvCache::clear, including the Self::alloc_id re-mint — every storage Arc is released, so a plan welded to them must be rebuilt.

pub fn truncate_to(&mut self, new_len: usize)

Shrink the cache to new_len cached positions — metadata-only, same Phase E.3.0 limitation and rationale as [KvCache:: truncate_to]: the underlying storages are not shrunk; the pre-allocated buffers’ trailing rows simply stop being read until overwritten by a later Op::WriteSlice at the same positions.

Deliberately preserves Self::alloc_id — a rewind is not a re-allocation. Same rule as KvCache::truncate_to.

Trait Implementations

impl KvRebindSource for LatentKvCache

fn alloc_id(&self) -> KvAllocId

Identity of this allocation. See KvAllocId.

fn rebind_location(&self) -> Option<DeviceLocation>

Where the storage lives, or None when that is not provable — which DecodeSession::rebind_kv treats as a refusal, never as a default.

fn rebind_slot( &self, layer: usize, which: usize, ) -> Option<Arc<RwLock<Storage>>>

Storage for slot which (0 or 1) of layer — the two halves of the plan’s kv_nodes[layer] pair, in the order the builder bound them.