Struct KvShape 

pub struct KvShape {
    pub n_layers: usize,
    pub layout: KvLayout,
    pub elem: KvElem,
    pub sliding: Option<SlidingWindow>,
}

The KV shape of a whole model: enough to price any context length.

Fields

n_layers: usize

layout: KvLayout

elem: KvElem

sliding: Option<SlidingWindow>

None = every layer keeps the full causal history.

Implementations

impl KvShape

pub fn from_config(config: &ModelConfig, elem: KvElem, chunk: usize) -> Self

Reads the shape off a config. chunk is the prefill chunk size the run will use (FERROX_CHUNKED_PREFILL, or 1 when prefill is token-at-a-time); it only ever matters for sliding layers.

Always produces a KvLayout::Gqa layout, because ModelConfig describes the generic GQA decoder – the MLA stacks carry their own hyperparameters (Deepseek2Hparams, MlaConfig) and should build their shape with KvShape::mla_expanded.

pub fn mla_expanded( n_layers: usize, n_heads: usize, qk_nope_head_dim: usize, qk_rope_head_dim: usize, v_head_dim: usize, elem: KvElem, ) -> Self

The shape a ferrox MLA decoder really allocates – see KvLayout::MlaExpanded.

pub fn sliding_layers(&self) -> usize

How many layers slide, given the alternating pattern.

pub fn full_attention_layers(&self) -> usize

Layers that keep the full causal history.

pub fn per_token_kv_bytes(&self) -> u64

The plan’s headline number: bytes one token costs across every layer, ignoring any sliding-window cap. Exact for f32/f16; for the block-quantized wires it is exact whenever a layer’s per-token element count is a multiple of the 32-element block (true for every real head-dim/kv-head combination), and rounds up otherwise.

pub fn marginal_per_token_bytes(&self) -> u64

Bytes each additional context token costs once the sliding layers have saturated: only the full-attention layers keep growing. This is the divisor KvBudget::max_context uses, and it is 0 for a model whose every layer slides – such a model’s KV is bounded no matter how long the context is.

pub fn kv_bytes_for_tokens(&self, tokens: usize) -> u64

Bytes one request’s KV costs at tokens of context, applying the sliding-window cap per layer class.

pub fn describe(&self) -> String

The sentence a user should be able to read and reproduce with a calculator.

Trait Implementations

impl Clone for KvShape

fn clone(&self) -> KvShape

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for KvShape

impl Debug for KvShape

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for KvShape

impl PartialEq for KvShape

fn eq(&self, other: &KvShape) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for KvShape