Struct MemoryBudget 

pub struct MemoryBudget {
    pub total_system_bytes: usize,
    pub available_bytes: usize,
    pub model_weight_bytes: usize,
    pub kv_cache_budget: usize,
    pub runtime_overhead: usize,
}

Memory budget estimation for a model deployment.

Fields

total_system_bytes: usize

Total system memory the user wants to allocate (bytes).

available_bytes: usize

Bytes available after subtracting model weights and runtime overhead.

model_weight_bytes: usize

Model weight footprint (bytes).

kv_cache_budget: usize

Budget specifically earmarked for KV cache (bytes).

runtime_overhead: usize

Estimated runtime overhead for buffers, activations, etc. (bytes).

Implementations

impl MemoryBudget

pub fn estimate( total_available_mb: usize, model_params: usize, bits_per_weight: f32, ) -> Self

Estimate budget.

• total_available_mb — how much RAM (MB) the user is willing to use.
• model_params — total number of parameters in the model.
• bits_per_weight — quantisation bits per weight (e.g. 1.125 for Q1_0).

pub fn max_context_length( &self, num_layers: usize, num_heads: usize, head_dim: usize, ) -> usize

Maximum context length that fits in the KV cache budget.

KV cache size per token = 2 (K+V) * num_layers * num_heads * head_dim * bytes_per_element. We use FP16 (2 bytes) as the default element type for estimation.

pub fn fits_context( &self, ctx_len: usize, num_layers: usize, num_heads: usize, head_dim: usize, ) -> bool

Whether a given context length fits in the KV cache budget.

pub fn summary(&self) -> String

Human-readable summary.

Trait Implementations

impl Clone for MemoryBudget

fn clone(&self) -> MemoryBudget

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MemoryBudget

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.