Expand description
KV cache primitives for a single sequence within a single transformer layer.
Stores one contiguous, pre-allocated buffer for all tokens’ keys and one
for all values (shape: max_tokens_per_seq * num_heads * hidden_dim_per_head
floats each), instead of one small heap allocation per token. The previous
design called resize_with on a Vec<KVCacheEntry> where every entry
independently allocated its own keys/values Vec<f32> — for the
default 8192-token max sequence length, a full-length initialize()
performed up to 16,384 separate small heap allocations. A single upfront
allocation eliminates that, keeps a sequence’s KV data in one contiguous
cache-friendly region, and enables read_range — a single batched read
over a token span, the shape attention actually needs, instead of N
separate per-token reads.
Reads and writes are guarded by an RwLock rather than a Mutex: real
attention reads happen far more often than writes (every position reads
all previous positions’ KV every decode step; only one write happens per
step), so letting reads proceed concurrently with each other matters.
This module has no current caller in runtime.rs — Ghostlink delegates
actual model execution to an external inference engine (llama-server /
Ollama, see ghost-link::native_engine) rather than computing attention
itself. It stays self-contained as a ready-to-use primitive for a future
local execution path, so it can evolve without affecting the stable
public API exported from lib.rs.
Structs§
- KVCache
Config - KVCache
Entry - Owned copy of one token’s cached key/value vectors, returned by
read_kv. Small and fixed-size (config.token_width()floats each) — not a copy of the whole cache. - Layer
KvCache - Cheap to clone (an
Arcaround the shared, lock-guarded buffer) — matches the shared-state pattern used elsewhere in this crate (e.g.ClusterState).