# skippy-cache TODO
## Completed In This PR
- Serving-path exact prefix cache is wired into local OpenAI serving:
`ResidentKv` and `KvRecurrent` policies restore before decode, record after
prefill, dedupe exact payloads, evict by entry/byte caps, and emit cache
telemetry. `FullState` remains a correctness/certification payload only; it
is not selected by production family policy.
- README benchmark evidence now has an explicit production-payload table.
Every reviewed family with a local full GGUF is benchmarked with either
`ResidentKv` or `KvRecurrent`; `FullState` rows are excluded from production
performance evidence.
- Benchmark evidence now uses matched single-request settings:
Skippy `--runtime-lane-count 1` and llama-server `--parallel 1`. Resident KV
rows report measured cache footprint when native KV-page export is available
and metadata-derived native KV footprint otherwise. The benchmark table has no
missing cache-size cells.
- DeepSeek3 package-backed cache strategy is certified for `ResidentKv` without
loading or merging the full 406.8 GB source GGUF. The gate covers real-input
`0..1`, real-upstream expert layer `3..4`, and synthetic-upstream late-layer
package stages `30..31` and `60..61`.
## DeepSeek3 Exact-State Certification
DeepSeek3 is classified as its own topology family and uses package-backed
`ResidentKv` as the accepted serving cache policy. We do not require a full GGUF
llama-server baseline for this family because the full layer set is too large
for the local baseline target.
Completed local package gates:
1. `0..1` with real token input: pass, `5.18x` vs stage recompute.
2. `3..4` with a real upstream `0..3` activation producer: pass, `3.76x` vs
stage recompute.
3. `30..31` with deterministic synthetic upstream activation: pass, `4.51x` vs
stage recompute.
4. `60..61` with deterministic synthetic upstream activation and output head:
pass, `2.03x` vs stage recompute.
Keep `q8_wire_validation` as untested until the exact package ref is certified
with q8 activation wire.
## Follow-Up Certification
- DeepSeek3 remains package-only for benchmark evidence. If a machine with
enough memory can run the monolithic full GGUF under llama-server, add that as
a separate baseline, but do not block package-backed serving or cache strategy
on that baseline.
- MiniMax M2.7 passes `ResidentKv`, and the exported recurrent component is
zero bytes for the tested GGUF. Add a multi-token continuation certification
and a compact recurrent-payload review before treating that as proof for every
MiniMax-style recurrent variant.
- Extend the production benchmark to longer prompts and multi-token generation.
The current smoke matrix shows dense-family parity/wins against llama-server
warm slots, but longer prompts and generation lengths should be tracked before
making release-level claims.
- Promote exact decoded-result caching from the correctness harness into serving:
cache full-prompt state/logits for exact repeated prompts, return the cached
first token without re-running the final prompt-token decode, and continue
normal decode only when `max_tokens > 1`.
## Cache Optimizations To Try
These are follow-on experiments from the cache literature pass. Keep them behind
feature/config flags until they have correctness tests, benchmark numbers, and
clear failure behavior.
1. Radix prefix index:
- Replace or augment flat exact-prefix lookup with a token radix tree.
- Return the longest exact prefix hit, not only full-prompt hits.
- Track model ref, tokenizer/template, topology, layer range, ABI, KV config,
and position config at each cacheable node.
- Measure lookup overhead and reuse rate against the current BLAKE3 exact-key
path.
- Expected outcome: repeated prompts with shared long prefixes hit earlier
prefixes instead of falling back to full recompute.
- Exit criteria: accept if longest-prefix reuse improves benchmark hit rate
or p50 latency by at least 10% with less than 1% lookup overhead; reject if
memory overhead or invalidation complexity makes flat exact keys simpler
and faster.
2. Page-aligned KV/state chunks:
- Store cache payloads in page-sized chunks aligned with llama/skippy KV
pages where possible.
- Deduplicate at the chunk/page layer instead of one monolithic state blob.
- Measure import/export latency, duplicate block rate, and memory
fragmentation.
- Expected outcome: less copying and better dedupe for prompts that share
prefix pages.
- Exit criteria: accept if physical cache bytes drop by at least 20% or
import/export latency drops by at least 10% without correctness fallout;
reject if page bookkeeping costs more than block-level BLAKE3 dedupe saves.
3. Hot/warm/cold cache tiers:
- Keep hot exact states resident in GPU or runtime memory.
- Keep warm deduped payloads in host memory.
- Optionally spill cold payloads to disk only for benchmark exploration; do
not put serialized disk restore on the serving hot path by default.
- Add per-model byte caps and eviction telemetry before enabling this in
normal serving.
- Expected outcome: high-value entries stay fast while total cache size is
bounded per model.
- Exit criteria: accept hot/warm tiers if p95 latency stays stable under
eviction pressure and byte caps are honored; keep disk cold tier
benchmark-only unless it beats recompute for large prefixes by at least 2x.
4. Async prefetch and staged restore:
- Begin importing the best prefix hit while OpenAI request parsing and route
setup continue.
- For split serving, prefetch downstream stage state before stage 0 finishes
frontend setup.
- Benchmark whether prefetch hides import latency or simply adds scheduling
contention.
- Expected outcome: restore time is overlapped with request setup and
downstream stage readiness.
- Exit criteria: accept if p50/p95 cache-hit latency improves by at least 10%
without increasing miss latency or lock contention; reject if prefetch makes
misses or concurrent requests noisier.
5. Batched cache movement:
- Batch export/import of KV pages, recurrent state, and full-state payloads
into fewer ABI calls.
- Prefer contiguous memory movement when the runtime can expose it.
- Measure CPU overhead and tail latency for concurrent lane restores.
- Expected outcome: fewer ABI crossings and better tail latency when many
lanes restore cache entries concurrently.
- Exit criteria: accept if cache-hit p95 improves by at least 10% or CPU time
per restore drops measurably; reject if batching adds latency for small
payloads or complicates error recovery.
6. Chunk-aware prefix reuse:
- Try chunked prefix matching for very long prompts so partial reuse can
happen at chunk boundaries.
- Validate that restored position/state is exactly equivalent before decode.
- Compare against the radix longest-prefix hit path.
- Expected outcome: very long prompts can reuse a safe chunk boundary even
when the full prefix is not cached.
- Exit criteria: accept only if correctness matches full recompute for every
tested family and long-context benchmarks beat radix-only lookup; reject if
boundary state is ambiguous or family-specific.
7. Cold-tier compression:
- Try lossless compression for cold exact payloads after BLAKE3 dedupe.
- Keep hot and warm cache hits uncompressed unless measurements show import
still wins.
- Record compression ratio, compression time, decompression time, and net hit
latency.
- Expected outcome: larger cold caches fit within the same byte budget
without affecting hot path latency.
- Exit criteria: accept only for cold entries if compressed restore still
beats recompute and saves at least 25% physical bytes; reject for hot/warm
entries unless decompression overhead is negligible.
8. Approximate KV compression and eviction:
- Explore H2O/SnapKV-style approximate retention only as a separate
non-exact mode.
- Never mix approximate entries into the exact cache namespace.
- Require output-quality benchmarks before considering serving use.
- Expected outcome: optional approximate mode may trade exactness for larger
effective context/cache capacity.
- Exit criteria: keep disabled unless quality benchmarks show acceptable
degradation for explicit opt-in workloads; reject as a default path because
exact cache correctness must remain binary.
9. Non-prefix document/RAG reuse:
- Investigate CacheBlend-style reuse for shared documents that appear inside
different prompts.
- Treat this as a separate cache mode because it is not exact token-prefix
restore.
- Start with correctness tooling only, not the OpenAI serving path.
- Expected outcome: document-heavy RAG workloads reuse shared context even
when the user prompt changes around it.
- Exit criteria: accept only as an explicit experimental mode if outputs are
validated against recompute and latency beats normal prefill on real RAG
traces; reject for the exact prefix serving path.
10. Native MLA/latent-state handling:
- For DeepSeek3 and other MLA families, identify whether the backend exposes
native latent KV/state or only expanded KV-compatible state.
- Certify exact replay with the actual exported representation before
enabling cache policy.
- Document whether payload economics differ from normal dense attention KV.
- Expected outcome: MLA families get the smallest exact replay payload the
backend can safely expose.
- Exit criteria: accept family policy only after state-handoff correctness
passes on representative GGUFs and payload/timing data is recorded; leave
DeepSeek3/MLA cache disabled if native latent state cannot be proven exact.