Skip to main content

Module kv_cache

Module kv_cache 

Source
Expand description

KV cache for autoregressive transformer generation.

Stores key and value projections for each layer across sequence positions. Designed for single-sequence generation (batch=1).

Structsยง

KVCache
KV cache for a single model.