Skip to main content

cache_bytes_per_token_for_layers

Function cache_bytes_per_token_for_layers 

Source
pub fn cache_bytes_per_token_for_layers(
    cfg: &ModelConfig,
    lo: usize,
    hi: usize,
) -> usize
Expand description

Context-linear cache bytes per token owned by layers in [lo, hi). PP admission uses the same layer ranges as Cache::new_ppn, so each device is charged for exactly the cache planes it allocates rather than for the aggregate model geometry.