Skip to main content

cache_bytes_per_token

Function cache_bytes_per_token 

Source
pub fn cache_bytes_per_token(cfg: &ModelConfig) -> usize
Expand description

Context-linear bytes allocated by one trunk cache token.

Fixed allocations (the 8-byte plane tail pads, len_d, recurrent state, and optional lazy buffers) are deliberately excluded. Admission adds their measured high-water residual as a request-independent activation term; multiplying this coefficient by the request’s own ctx_cap exactly mirrors the context-scaled allocations in Cache::new_inner.