Struct KvCacheConfig 

pub struct KvCacheConfig {
    pub num_hidden_layers: u32,
    pub num_key_value_heads: u32,
    pub head_dim: u32,
}

The three model-config fields the KV-cache formula actually needs.

These map onto num_hidden_layers, num_key_value_heads and head_dim in a Hugging Face config.json.

Fields

num_hidden_layers: u32

num_hidden_layers — every layer keeps its own K and V tensors.

num_key_value_heads: u32

num_key_value_heads — the GQA key-value head count, not the query head count.

head_dim: u32

head_dim — the per-head dimension of the key and value vectors.

Implementations

impl KvCacheConfig

pub fn new( num_hidden_layers: u32, num_key_value_heads: u32, head_dim: u32, ) -> Result<Self, KvError>

Builds a config, refusing any zero dimension.

Errors

KvError::ZeroDimension naming the offending field.

pub fn bytes_per_token(self, precision: KvPrecision) -> f64

Bytes the cache grows by for one additional token in one sequence.

This is the number that decides whether cache size or weight size dominates, and it is independent of context length and batch size.

pub fn total_bytes( self, precision: KvPrecision, context_length: u64, batch_size: u32, ) -> Result<f64, KvError>

Total cache bytes for context_length tokens across batch_size sequences.

Errors

KvError::ZeroWorkload if either argument is zero — a zero-token or zero-sequence workload is a caller mistake, not a zero-byte answer.

pub fn total_bytes_u64( self, precision: KvPrecision, context_length: u64, batch_size: u32, ) -> Result<u64, KvError>

The same total, rounded down to whole bytes.

Errors

As Self::total_bytes, plus KvError::Overflow when the result exceeds u64::MAX bytes.

pub fn max_context( self, precision: KvPrecision, budget_bytes: f64, batch_size: u32, ) -> Result<u64, KvError>

The longest context that fits budget_bytes, at this precision and batch size.

Rounds down: the returned length is guaranteed to fit. Returns 0 when not even one token fits, which is a real answer and not an error.

Errors

KvError::InvalidBudget for a negative or non-finite budget, and KvError::ZeroWorkload for a zero batch size.

pub fn gqa_overstatement(self, num_attention_heads: u32) -> Option<f64>

The factor by which using num_attention_heads instead of num_key_value_heads would overstate the cache.

This is the GQA group size. It is 1.0 for a model without GQA, and the full head count for multi-query attention. Returns None if num_attention_heads is zero.

Trait Implementations

impl Clone for KvCacheConfig

fn clone(&self) -> KvCacheConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for KvCacheConfig

impl Debug for KvCacheConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for KvCacheConfig

impl PartialEq for KvCacheConfig

fn eq(&self, other: &KvCacheConfig) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for KvCacheConfig