Skip to main content

new_cache

Function new_cache 

Source
pub fn new_cache(
    e: &Engine,
    cfg: &ModelConfig,
    max_ctx: usize,
) -> Result<Cache, Box<dyn Error>>
Expand description

Stage-owned cache allocation door: when the ppN door is open AND MEMRA_PP_DEVICES is set (placement plumbing), each layer’s cache is allocated by its OWNING stage’s engine — on one device this is byte-for-byte today’s allocation (gated); cross-device it puts each stage’s KV on that stage’s HBM. Door shut or devices unset: plain Cache::new (zero behavior change). Trailing MTP/NextN layers (beyond the trunk) map to the LAST stage.