skippy-cache

skippy-cache owns the cache model for staged serving. It does not talk to llama.cpp, open sockets, route OpenAI requests, or plan topology. Those responsibilities stay in skippy-server, skippy-runtime, and mesh.

The crate answers cache questions:

• what exact prefix identity should be used for this stage?
• which prefix lengths should be looked up or recorded?
• which resident cache entry should be evicted?
• how should exact state payload bytes be represented and deduplicated?

The cache is deliberately exact. It does not guess that two prompts are "similar enough"; it reuses runtime state only when the model, topology, stage, layer range, runtime layout, position, and token prefix all match.

Boundaries

flowchart LR
    OpenAI["openai-frontend"] --> Server["skippy-server"]
    Server --> Runtime["skippy-runtime"]
    Runtime --> ABI["llama.cpp stage ABI"]

    Server --> Cache["skippy-cache"]
    Cache --> Identity["prefix identity"]
    Cache --> Resident["resident prefix / activation caches"]
    Cache --> Payload["exact state payloads"]
    Payload --> Blocks["BLAKE3 block store"]

    Cache -. no runtime calls .- Runtime

skippy-server remains the adapter. It turns protocol messages into cache lookups, performs the runtime save/restore/import/export calls, and records telemetry. skippy-cache only owns pure data structures and policies.

Prefix Identity

Prefix identity is exact. A hit is valid only for the same:

• model id
• topology id
• stage id and stage index
• layer range
• runtime ABI schema
• KV layout and dtype
• context/position configuration
• token start and token ids

The identity hash uses BLAKE3 over these fields. The short page id is derived from the hash and is suitable for logs and resident-cache maps.

flowchart TD
    Tokens["token ids + token_start"] --> Hash["BLAKE3 prefix hash"]
    Stage["model/topology/stage/layers"] --> Hash
    Runtime["ABI + layout + dtype + ctx"] --> Hash
    Hash --> Page["page id"]
    Page --> Lookup["resident/exact-state lookup"]

Resident Cache Flow

The current resident path keeps reusable state inside the live llama.cpp session. skippy-cache chooses candidates and tracks entries; skippy-server does the native sequence copy/drop calls.

sequenceDiagram
    participant S as skippy-server
    participant C as skippy-cache
    participant R as skippy-runtime

    S->>C: candidate prefix identities
    C-->>S: longest-first lookup candidates
    S->>C: lookup resident entry
    alt hit
        C-->>S: cache seq id
        S->>R: restore resident prefix
        S->>S: compute only suffix or forward cached activation
    else miss
        S->>R: prefill normally
        S->>C: allocate record slot
        C-->>S: cache seq id + evictions
        S->>R: drop evicted seq ids
        S->>R: save resident prefix
        S->>C: commit record
    end

Activation frames are cached separately by act:{page_id}:w{activation_width}. In a split topology, an upstream stage may reuse both its resident prefix and the activation frame it would have forwarded downstream.

For dense models this is the production fast path. It is intentionally close to the llama-server slot contract: keep usable KV resident in the runtime and continue from it. The difference is that Skippy gives the resident state a backend-neutral prefix identity, keeps hot lane prefixes borrowed in place when possible, and can reuse the same prefix across Rust-owned serving, staged execution, and activation forwarding.

Exact State Payloads

The payload module models the portable exact-cache path we need for recurrent/stateful families:

Large payloads are split into 1 MiB BLAKE3-addressed blocks. Repeated blocks are stored once, so capacity is accounted by physical bytes as well as logical payload bytes.

KvRecurrent is the production method for hybrid/recurrent families. It moves attention KV together with recurrent/SSM state, because KV alone is not the full continuation state for those models. That is where the largest wins come from: llama-server warm slots still have to reprocess the recurrent prefix in many request shapes, while Skippy can restore the exact compact state and decode.

flowchart LR
    Payload["KV + recurrent payload"] --> Split["1 MiB chunks"]
    Split --> H1["BLAKE3 block A"]
    Split --> H2["BLAKE3 block B"]
    Split --> H3["BLAKE3 block A reused"]
    H1 --> Store["block store"]
    H2 --> Store
    H3 -. refcount .-> Store
    Store --> Stats["logical / physical / saved bytes"]

Family Cache Policy

Cache support is family-dependent because different model families have different continuation state.

The serving integration currently scans GGUF tensor names and disables the KV-only resident path when recurrent/stateful tensors are present, including .ssm, ssm_, time_mix, recurrent, and rwkv. That guard prevents a hybrid model from importing attention KV while missing the recurrent state that changes the next token.

The intended recurrent/stateful serving path is:

1. lookup exact prefix identity
2. import/restore KvRecurrent
3. recompute safely on any miss or incompatible payload
4. record a new exact payload after successful prefill

The OpenAI serving path now uses the same cache policy. A cache-enabled stage selects one payload shape:

• ResidentKv for certified dense families where resident llama.cpp sequence copy is enough
• KvRecurrent for hybrid/recurrent families where attention KV and recurrent state must move together

FullState is intentionally not selected by the production family policy. It is kept as a correctness/certification tool so a family can first prove exact restore behavior before we design and promote a compact serving payload.

Why Skippy Cache Beats llama-server

llama-server slots are very good at one thing: keeping a prompt prefix warm inside a server slot. Skippy preserves that behavior for dense single-stage serving, then extends it in several important ways.

The headline pattern in the benchmark table follows from those methods:

• Hybrid/recurrent families show the largest wins because KvRecurrent restores state llama-server otherwise has to regenerate.
• Dense families show parity to moderate wins because ResidentKv keeps the same warm-slot advantage while avoiding extra serving-path overhead.
• DeepSeek3 is package-only evidence: it proves the selected cache method for a materialized stage, not a full-GGUF llama-server comparison.

The cache does not claim fuzzy semantic reuse, approximate prefix matching, or FullState production speedups. FullState remains a certification tool only.

flowchart LR
    Request["request tokens"] --> Identity["exact prefix identity"]
    Identity --> Hit{"cache hit?"}
    Hit -- dense --> Resident["ResidentKv: borrow resident runtime KV"]
    Hit -- recurrent --> Recurrent["KvRecurrent: restore KV + recurrent state"]
    Hit -- split --> Activation["activation frame reuse"]
    Hit -- miss --> Prefill["normal prefill"]
    Prefill --> Record["record exact state"]
    Record --> Dedupe["BLAKE3 block dedupe + capacity accounting"]

sequenceDiagram
    participant O as openai-frontend
    participant S as skippy-server
    participant C as skippy-cache
    participant R as skippy-runtime

    O->>S: request
    S->>C: longest-prefix identities
    alt exact hit
        C-->>S: KvRecurrent payload
        S->>R: import state into lane
        S->>R: decode next token
    else resident hit
        C-->>S: resident seq id
        S->>R: restore resident prefix
        S->>R: decode next token
    else miss
        S->>R: prefill
        S->>R: export selected payload
        S->>C: dedupe/store and evict by cap
        S->>R: decode next token
    end

Benchmark Evidence

README performance claims must be backed by correctness runs and comparable llama-server baselines. Rows marked untested are intentionally not promoted as default evidence yet.

Reproduce the family evidence and README tables with:

evals/skippy-cache-family-bench.sh /tmp/skippy-cache-family-bench

For a faster rerun after building locally:

SKIPPY_CACHE_SKIP_BUILD=1 evals/skippy-cache-family-bench.sh /tmp/skippy-cache-family-bench

Local correctness evidence below was collected on the same machine with n_predict = 1, n_gpu_layers = -1, Skippy --runtime-lane-count 1, and llama-server --parallel 1. The payload column is the production serving payload, not an experimental fallback. Results use median warm-hit latency from matched repeated prompts. The full-GGUF table uses the same 128-token requested prefix and one generated token for every row so each Skippy result is compared against the matching llama-server workload.

Cache bytes is serialized payload size for KvRecurrent and measured native KV-page footprint for ResidentKv when the runtime can expose it. Rows marked metadata-derived use the same llama.cpp KV dimensions from GGUF metadata: active KV layers, KV heads, key/value head lengths, SWA pattern, shared-KV layers, token count, and f16 KV element size.

Full-GGUF llama-server vs Skippy

Rows are ordered so related runtime/cache families appear next to each other.

Use-Case Benchmark Matrix

This matrix uses one Hugging Face-sourced representative prompt per use case, the same 128 requested prefix tokens, one generated token, Skippy --runtime-lane-count 1, llama-server --parallel 1, and the same full-GGUF family set as the table above. Values are Skippy warm-hit latency speedup over llama-server warm-cache latency. DeepSeek3 stays in the package-only section because there is no practical local full-GGUF llama-server baseline for that artifact.

Prompt sources are checked in at evals/skippy-usecase-corpus.json with source dataset metadata:

Package-Only Giant Models

These rows validate cache strategy for models where a full llama-server baseline is not operationally useful because monolithic residency is too large.

DeepSeek3 uses ResidentKv for package-backed serving. The local gate verifies the package stage cache without requiring a monolithic full-GGUF baseline: 0..1 passed with real token input, 3..4 passed with a real upstream 0..3 activation producer, and late layers 30..31 plus 60..61 passed with deterministic synthetic upstream activations so only the owned stage range had to be materialized.

state-handoff reports distinguish behavioral exactness from byte-stable state re-export. status = pass means the restored cache state produced the same next token/output and repeat hits matched. roundtrip_state_matches = false means a state exported after import was not byte-for-byte identical; that is a canonicalization/deduplication diagnostic, not a serving correctness failure. Raw reports from this run are under /tmp/skippy-cache-production-bench-track1-apples-p128.

Current Performance Read

With llama-server-compatible thread defaults, hot-lane resident prefix reuse, and the production borrowed-prefix path, Skippy is at parity or ahead for every locally available family in the normalized single-stage matrix:

Before claiming a new family is faster than llama-server, run the reproducible benchmark wrapper and keep the matched conditions fixed: same GGUF, backend, context size, prompt tokens, generated tokens, slot/parallel settings, and warm-cache policy. The README table is a serving-correctness table first; performance wins are only promoted for rows whose matched speedup is above 1.0x.

Module Map

Testing

The crate is intentionally lightweight. It should test without building or linking llama.cpp:

cargo test -p skippy-cache --lib

Supported-family cache safety is smoked through the topology capability registry:

cargo test -p skippy-topology reviewed_supported_families_smoke --lib

That smoke does not execute model files. It verifies that every reviewed supported family can be inferred, planned with f16 serving wire, and emits the expected recurrent/sticky, q8 validation, and sideband policy signals.

Server/runtime integration tests belong in skippy-server, skippy-runtime, and skippy-correctness.