Module cache 

The dual cache lives in the shared memra-kv crate (Phase D extraction); this re-export keeps every crate::cache:: / memra_engine::cache:: path unchanged.

Structs

Cache

CacheSnapshot

Snapshot of the dual cache taken BEFORE a spec-decode draft+verify round (MTP-PLAN §C/§D.4).

DflashTapSink

See Cache::dflash_taps. Armed per forward by the dflash round (t = that forward’s row count); the trunk writes tap slot s of row r at buf[rn_tapshidden + s*hidden ..].

KvLayer

Per-full-attn-layer growing KV cache, resident on GPU. QUANTIZED (KVQUANT-PLAN §B): K stored q8_0 (34 B/32 elem), V stored q5_1 (24 B/32 elem). Per-token byte layout keeps the [token, kv_head, dim] element order so a 32-block never straddles a head (assert head_dim%32==0). Element-within-token index = kv_head*head_dim + d; block = idx/32; lane = idx%32.

KvRing

RecurLayer

Per-linear-attn-layer fixed recurrent state. conv_state and ssm_state are BOTH kept RESIDENT on GPU — the conv ring assemble + roll runs on-device (conv_assemble_and_roll), so there is no per-step dtoh/htod for either.

Enums

KvRingAppend

Constants

PRIME_CHUNK_MAX_TOKENS

With the SWA-ring door open, prime_chunk_tokens caps every legal chunk at this bound. The ring carries one whole maximum-size prime chunk in addition to the reader’s window.

Statics

KV_FP8_FORCE

Per-model FP8-KV door (-1 = unset → env/default off; 0 = off; 1 = on). Set at qwen model load: the 2026-07-12 arc closed per-model — 9B +0.7-4% scaling with depth, 27B flat (weight-bound), 35B −2% (fp8 format-gates its v3 dp4a lane off). Explicit MEMRA_KV_FP8 wins. 9B adoption attempt REVERTED by measurement 2026-07-29 (−1% at 12k on the then-current build) — loaders currently store 0.

Traits

KvDev

The 7 device ops the cache needs — nothing more. Implemented by the engine (and by any future backend); all ops are stream-ordered on the implementor’s worker stream.

Functions

cache_bytes_per_token

Context-linear bytes allocated by one trunk cache token.

cache_bytes_per_token_for_layers

Context-linear cache bytes per token owned by layers in [lo, hi). PP admission uses the same layer ranges as Cache::new_ppn, so each device is charged for exactly the cache planes it allocates rather than for the aggregate model geometry.

cache_ring_bytes_per_token

Portion of cache_bytes_per_token whose physical row count is capped by the Step35 SWA ring. Zero with the flag off and for every non-Step35 architecture.

cache_ring_bytes_per_token_for_layers

Ring-capped portion of cache_bytes_per_token_for_layers for [lo, hi).

cache_ring_row_cap

Physical row cap shared by the Step35 SWA trunk and MTP scratch; zero when no ring is active.

gkv_on

FP8-GLOBALS switch (MEMRA_GEMMA_GKV, default ON): gemma global (hd512) layers keep their KV in e4m3 — the dequant-latency arc (HANDOVER). Windowed layers stay q8_0/q5_1.

kv_blk_bytes

Per-32-element block bytes for the selected (K, V) formats.

kv_cache_formats

Env-selected KV cache formats (MEMRA_KV_K / MEMRA_KV_V). The engine’s flash-fatbin router and the cache sizing below MUST agree — both read this one function.

kv_fp8_on

QWEN FP8-KV switch (MEMRA_KV_FP8 explicit; else the per-model KV_FP8_FORCE door set at model load; else OFF). Non-gemma full-attn layers hold e4m3 K/V via the kf8vf8 module.

swa_ring_on

Step35 SWA-ring experiment (default OFF). The first cut is deliberately architecture-scoped: Gemma4’s row-0-addressed window kernels cannot consume a rebased ring view.

swa_ring_rows

Physical rows required by the Step35 SWA reader contract. Prime starts at (base_len - (window - 1)) & !31, so at most 31 masked rows precede the live window.

wkv_on

FP8-WINDOWED switch (MEMRA_GEMMA_WKV; serving-mode default): SPEC serving (MEMRA_DRAFT set) -> OFF, plain -> ON — the acceptance-vs-depth record lives on the engine-side history of Engine::wkv_on (git). Explicit env always wins.