Skip to main content

plan_for

Function plan_for 

Source
pub fn plan_for(
    weight_bytes: u64,
    available: Option<u64>,
    headroom_bytes: u64,
    min_cache_bytes: u64,
) -> FitPlan
Expand description

Decide whether streaming is needed, given what the model weighs and what the host has.

The bar is deliberately not “does it fit exactly”. A model that fills memory to the last byte will thrash the page cache and leave nothing for the KV cache, so headroom_bytes is subtracted first.

available of None means the platform would not say. That resolves to Resident, NOT to streaming: guessing that a machine is short on memory would silently put every user on the slow path on any platform without a probe.