Module kv_cache

Structs§

ConvHistory: Zero-allocation flat ring buffer storing recent convolution state snapshots for speculative decoding rollback.
InferenceState: Inference state across all layers.
KvCacheConfig: Configuration for the KV prefix cache.
KvPrefixCache: Two-tier KV prefix cache: warm (memory) + cold (disk via FlatBuffers).
ScratchBuffers: Pre-allocated scratch buffers reused across layers and tokens.
StateSnapshot: Snapshot of model KV + conv state after prefilling a token sequence. Backend-agnostic: stores raw bytes that the backend knows how to restore.

KvCompression: KV cache compression mode. Passed to InferenceState::from_config_with_compression (or via GenerateConfig::kv_compression) — that single call sets up everything TurboQuant needs: the per-layer rotation states, the compressed key/value caches, and the scratch buffers. No separate enable_turboquant call on the model is required.
LayerSnapshot
LayerState
SemanticBoundaryKind: Classification of semantic boundaries in agentic and multimodal workflows.

CONV_HISTORY_CAPACITY: Per-layer inference state. Capacity for the Conv rollback ring buffer (in number of tokens).

model_fingerprint: Compute a fingerprint for a model configuration. Two models with different fingerprints have incompatible KV cache layouts. Callers should pass a model_id that uniquely identifies the specific model weights (e.g. a hash of the GGUF file or the model name from metadata), so different models with the same architecture don’t share cache entries.