Skip to main content

net_cache_budget_bytes

Function net_cache_budget_bytes 

Source
pub fn net_cache_budget_bytes(
    memory_ratio: f64,
    baseline_free_bytes: u64,
    weights_bytes: u64,
    fixed_cache_bytes: u64,
) -> i64
Expand description

The VRAM a rebuild may spend: a fraction of what was free before the weights loaded, minus the weights, minus whatever the pools need unconditionally.

The fraction is not padding for its own sake – it is the room the activations, the workspace, and the captured graphs occupy, none of which is in weights_bytes. Spending it produces an allocation failure at the first long prompt rather than at startup.

Signed on purpose: an over-committed deployment gets a negative budget, which every consumer below then refuses, rather than a wrapped-around enormous one.