Module cache 

The dual cache lives in the shared memra-kv crate (Phase D extraction); this re-export keeps every crate::cache:: / memra_engine::cache:: path unchanged.

Structs

Cache

CacheSnapshot

Snapshot of the dual cache taken BEFORE a spec-decode draft+verify round (MTP-PLAN §C/§D.4).

DflashTapSink

See Cache::dflash_taps. Armed per forward by the dflash round (t = that forward’s row count); the trunk writes tap slot s of row r at buf[rn_tapshidden + s*hidden ..].

KvLayer

Per-full-attn-layer growing KV cache, resident on GPU. QUANTIZED (KVQUANT-PLAN §B): K stored q8_0 (34 B/32 elem), V stored q5_1 (24 B/32 elem). Per-token byte layout keeps the [token, kv_head, dim] element order so a 32-block never straddles a head (assert head_dim%32==0). Element-within-token index = kv_head*head_dim + d; block = idx/32; lane = idx%32.

RecurLayer

Per-linear-attn-layer fixed recurrent state. conv_state and ssm_state are BOTH kept RESIDENT on GPU — the conv ring assemble + roll runs on-device (conv_assemble_and_roll), so there is no per-step dtoh/htod for either.

Statics

KV_FP8_FORCE

Per-model FP8-KV door (-1 = unset → env/default off; 0 = off; 1 = on). Set at qwen model load: the 2026-07-12 arc closed per-model — 9B +0.7-4% scaling with depth, 27B flat (weight-bound), 35B −2% (fp8 format-gates its v3 dp4a lane off). Explicit MEMRA_KV_FP8 wins. 9B adoption attempt REVERTED by measurement 2026-07-29 (−1% at 12k on the then-current build) — loaders currently store 0.

Traits

KvDev

The 7 device ops the cache needs — nothing more. Implemented by the engine (and by any future backend); all ops are stream-ordered on the implementor’s worker stream.

Functions

gkv_on

FP8-GLOBALS switch (MEMRA_GEMMA_GKV, default ON): gemma global (hd512) layers keep their KV in e4m3 — the dequant-latency arc (HANDOVER). Windowed layers stay q8_0/q5_1.

kv_blk_bytes

Per-32-element block bytes for the selected (K, V) formats.

kv_cache_formats

Env-selected KV cache formats (MEMRA_KV_K / MEMRA_KV_V). The engine’s flash-fatbin router and the cache sizing below MUST agree — both read this one function.

kv_fp8_on

QWEN FP8-KV switch (MEMRA_KV_FP8 explicit; else the per-model KV_FP8_FORCE door set at model load; else OFF). Non-gemma full-attn layers hold e4m3 K/V via the kf8vf8 module.

wkv_on

FP8-WINDOWED switch (MEMRA_GEMMA_WKV; serving-mode default): SPEC serving (MEMRA_DRAFT set) -> OFF, plain -> ON — the acceptance-vs-depth record lives on the engine-side history of Engine::wkv_on (git). Explicit env always wins.