pub fn plan_for(
weight_bytes: u64,
available: Option<u64>,
headroom_bytes: u64,
min_cache_bytes: u64,
) -> FitPlanExpand description
Decide whether streaming is needed, given what the model weighs and what the host has.
The bar is deliberately not “does it fit exactly”. A model that
fills memory to the last byte will thrash the page cache and leave
nothing for the KV cache, so headroom_bytes is subtracted first.
available of None means the platform would not say. That
resolves to Resident, NOT to streaming: guessing that a machine is
short on memory would silently put every user on the slow path on
any platform without a probe.