Struct KvShape 

pub struct KvShape {
    pub n_layers: usize,
    pub layout: KvLayout,
    pub elem: KvElem,
}

The KV shape of a whole model: enough to price any context length.

How big one position is, times how many layers. How many positions each of those layers still HOLDS is KvResidency, and it is a separate value because it is a property of the run rather than of the model: by default every layer keeps every position, and behind FERROX_KV_WINDOW a windowed layer does not (#61).

That is a statement about the STORES this engine allocates, not about the architectures it runs – see the module doc, and the two tests that measure real ferrox_core::cache::KvCaches rather than restating this multiplication.

Fields

n_layers: usize

layout: KvLayout

elem: KvElem

Implementations

impl KvShape

pub fn from_config(config: &ModelConfig, elem: KvElem) -> Self

Reads the shape off a config.

config.sliding_window / config.swa_pattern are deliberately NOT read HERE: they describe what attention reads, and this module prices what the store keeps. The default store keeps everything (#33), so a windowed layer costs exactly what a full-attention one does. When a run evicts, that is what KvResidency::from_config is for – and it reaches the window through the same KvWindowPolicy the decoder evicts with, not by reading those two fields a second time.

Always produces a KvLayout::Gqa layout, because ModelConfig describes the generic GQA decoder – the MLA stacks carry their own hyperparameters (Deepseek2Hparams, MlaConfig) and should build their shape with KvShape::mla_expanded.

pub fn mla_expanded( n_layers: usize, n_heads: usize, qk_nope_head_dim: usize, qk_rope_head_dim: usize, v_head_dim: usize, elem: KvElem, ) -> Self

The shape a ferrox MLA decoder really allocates – see KvLayout::MlaExpanded.

pub fn per_token_kv_bytes(&self) -> u64

The plan’s headline number, and the only per-token number there is: bytes one token costs across every layer. Exact for f32/f16; for the block-quantized wires it is exact whenever a layer’s per-token element count is a multiple of the 32-element block (true for every real head-dim/kv-head combination), and rounds up otherwise.

This is also the divisor KvBudget::max_context uses. There is no separate “marginal” number any more: a marginal cost below the per-token cost would mean some layer stops growing, and none does.

pub fn kv_bytes_for_tokens(&self, tokens: usize) -> u64

Bytes one request’s KV costs at tokens of context.

pub fn resident_kv_bytes_for_tokens( &self, tokens: usize, residency: &KvResidency, ) -> u64

Bytes one request’s KV costs at tokens of context when the stores EVICT behind a window (#61 step 2), once every layer has been through – the number a measurement of the caches finds.

The row counts come from KvWindow::rows_after, which is the store’s own rule and not a restatement of it: KvCache calls the same function to decide what to drop. Equal to Self::kv_bytes_for_tokens when residency keeps everything, which is what the default policy produces and what a test below asserts rather than assumes.

Self::peak_kv_bytes_for_tokens is the number to ADMIT on; this one is smaller, and the difference is prefill.

pub fn peak_kv_bytes_for_tokens( &self, tokens: usize, residency: &KvResidency, ) -> u64

The number an admission decision must use: the resting cost, plus the one layer that is still mid-prefill.

Decoder::forward_batch writes a whole prompt into layer l’s cache, attends over it, and only then hands the rows behind the window back – before layer l + 1 allocates any. So a long prompt costs ONE windowed layer’s full history at a time rather than every windowed layer’s at once, and that transient is real memory that has to be budgeted for. Charging only the resting number would be #33 in the other direction: an admitted request whose peak exceeds the estimate arrives as an OOM.

The extra term is the largest single windowed layer’s shortfall, because layers are prefilled one at a time.

pub fn describe(&self) -> String

The sentence a user should be able to read and reproduce with a calculator.

Trait Implementations

impl Clone for KvShape

fn clone(&self) -> KvShape

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for KvShape

impl Debug for KvShape

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for KvShape

impl PartialEq for KvShape

fn eq(&self, other: &KvShape) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for KvShape