Module kv_cache 

KV cache for efficient autoregressive generation KV Cache for efficient autoregressive generation

This module provides a pre-allocated CPU-focused KV cache that stores computed Key and Value tensors to avoid recomputation during text generation.

How it works

During autoregressive generation, each transformer layer computes Key and Value tensors that are reused when generating subsequent tokens. Without caching, the model must recompute K,V for all previous tokens on each generation step, leading to O(n^2) complexity.

With KV caching:

• Prefill: Process entire prompt, cache K,V for each layer
• Decode: For each new token, only compute K,V for that token and append to cached values using slice_set (no allocation!)

This reduces generation to O(n) complexity, providing 50-100x speedup.

Structs

KVCache

Full model KV cache containing all layers

LayerKVCache

Per-layer KV cache storing Key and Value tensors with pre-allocated buffers

Constants

DEFAULT_MAX_SEQ_LEN

Default maximum sequence length for pre-allocation