Module cache 

The dual cache lives in the shared memra-kv crate (Phase D extraction); this re-export keeps every crate::cache:: / memra_engine::cache:: path unchanged.

Structs

Cache

CacheSnapshot

Snapshot of the dual cache taken BEFORE a spec-decode draft+verify round (MTP-PLAN §C/§D.4).

DflashTapSink

See Cache::dflash_taps. Armed per forward by the dflash round (t = that forward’s row count); the trunk writes tap slot s of row r at buf[rn_tapshidden + s*hidden ..].

HcTapSink

See Cache::hc_taps. Armed per walk by the glm5 DFlash2 draft source; the hc trunk writes the CONTRACTED (stream-mean) completed output of tapped layer layer_ids[s] for walk row r at rows[(base + r) * n_taps * hidden + s * hidden ..][..hidden] — the drafter fc’s input layout, measured by the dflash2 probe’s capture seam (research/glm53-flash-bringup-20260827/dflash2-probe-20260829/: stream-mean of the completed layer output == the SGLang glm5_next hc_contract aux-hidden definition).

KvLayer

Per-full-attn-layer growing KV cache, resident on GPU. QUANTIZED (KVQUANT-PLAN §B): K stored q8_0 (34 B/32 elem), V stored q5_1 (24 B/32 elem). Per-token byte layout keeps the [token, kv_head, dim] element order so a 32-block never straddles a head (assert head_dim%32==0). Element-within-token index = kv_head*head_dim + d; block = idx/32; lane = idx%32.

KvRing

LatentKvLayer

Per-MLA-layer latent KV plane (DESIGN.md §3.2). ONE row per token, width elements wide, where width == StatePlan::LatentKvCache { width } == kv_lora_rank + rope_head_dim: row = [ rmsnorm(c_kv) : kv_rank | rope(k_pe) : d_rope ] There is NO V plane — V is the FIRST kv_rank elements of the SAME row, and every query head streams that one row (MQA). NoPE models (glm5_next, rope_head_dim 0) have width == kv_rank and no k_pe tail.

LatentPlaneSnapshot

One MLA/DSA layer’s captured latent-plane state: everything mla_attn_cached + mla_kpool_indices need to continue as if the destination session had primed the prefix itself (lane/glm5-prefix-latent, 2026-08-30; design in research/glm5-prefix-latent-20260830/DESIGN.md).

LatentTailCapture

The generation-destroyed slice of one latent layer’s BOUNDARY state, captured EAGERLY at a spec session’s prompt boundary (lane/glm5-prefix-latent2, 2026-09-01) so a DEFERRED prefix publication can be completed later against the live plane:

RecurLayer

Per-linear-attn-layer fixed recurrent state. conv_state and ssm_state are BOTH kept RESIDENT on GPU — the conv ring assemble + roll runs on-device (conv_assemble_and_roll), so there is no per-step dtoh/htod for either.

ResidentTpKvCache

ResidentTpKvCacheRank

TpKvAppendPlan

TpKvGrowPlan

TpKvRankAllocationShape

Exact allocation geometry for one rank of a tensor-parallel KV sidecar.

TpKvTransaction

Enums

KvRingAppend

Constants

INDEX_RING_WORKING_ROWS

WORKING-SET rows the DSA indexer TAIL RING books by default.

PRIME_CHUNK_MAX_TOKENS

With the SWA-ring door open, prime_chunk_tokens caps every legal chunk at this bound. The ring carries one whole maximum-size prime chunk in addition to the reader’s window.

SWA_REWIND_SLACK_ROWS

Rows a ring-backed plane keeps BELOW the aligned window start so a backward rewind stays legal.

Statics

KV_FP8_FORCE

Per-model FP8-KV door (-1 = unset → env/default off; 0 = off; 1 = on). Set at qwen model load: the 2026-07-12 arc closed per-model — 9B +0.7-4% scaling with depth, 27B flat (weight-bound), 35B −2% (fp8 format-gates its v3 dp4a lane off). Explicit MEMRA_KV_FP8 wins. 9B adoption attempt REVERTED by measurement 2026-07-29 (−1% at 12k on the then-current build) — loaders currently store 0.

Traits

KvDev

The 7 device ops the cache needs — nothing more. Implemented by the engine (and by any future backend); all ops are stream-ordered on the implementor’s worker stream.

Functions

cache_bytes_per_token

Context-linear bytes allocated by one trunk cache token.

cache_bytes_per_token_for_layers

Context-linear cache bytes per token owned by layers in [lo, hi). PP admission uses the same layer ranges as Cache::new_ppn, so each device is charged for exactly the cache planes it allocates rather than for the aggregate model geometry.

cache_bytes_per_token_for_plan

cache_ring_bytes_per_token

Portion of cache_bytes_per_token whose physical row count is capped by the Step35 SWA ring. Zero with the flag off and for every non-Step35 architecture.

cache_ring_bytes_per_token_for_layers

Ring-capped portion of cache_bytes_per_token_for_layers for [lo, hi).

cache_ring_bytes_per_token_for_plan

cache_ring_row_cap

Physical row cap shared by the Step35 SWA trunk and MTP scratch; zero when no ring is active.

cache_ring_row_cap_for_plan

gkv_on

FP8-GLOBALS switch (MEMRA_GEMMA_GKV, default ON): gemma global (hd512) layers keep their KV in e4m3 — the dequant-latency arc (HANDOVER). Windowed layers stay q8_0/q5_1.

index_plane_physical_row

Physical row of absolute row abs in an indexer state plane. ring_rows == 0 is the flat plane (absolute addressing); otherwise the EFFECTIVE ring is ring_rows rounded down to a whole number of pools, exactly the engine’s own rounding (mla_kpool_indices), because a ring that is not a multiple of pool would split a pool across the wrap.

index_ring_default_rows

PHYSICAL rows the SHIPPED DEFAULT derivation books for max_ctx: no MEMRA_DSA_INDEX_RING override. Pure, so the sizing gate can assert on it without racing another test’s environment. This is the one function the sizing gate calls.

index_ring_rows

DSA k-pool indexer TAIL RING sizing (MEMRA_DSA_INDEX_RING, default ON, see docs/FLAGS.md). Unparseable values are treated as unset. MEMRA_PRIME_CHUNK is NO LONGER READ HERE: the ring is drained inside the call, so no prefill chunk discipline can size it or break it.

index_ring_rows_for

Physical rows of the DSA k-pool indexer state plane when it is a TAIL RING, or None to keep the flat max_ctx-row plane. PURE: the env read is index_ring_rows.

index_ring_take

Rows of remaining the indexer may append to the tail ring BEFORE the pool-key build has to drain it, or None when the rows this call still owes an unbuilt pool are already lapped.

kv_blk_bytes

Per-32-element block bytes for the selected (K, V) formats.

kv_cache_formats

Env-selected KV cache formats (MEMRA_KV_K / MEMRA_KV_V). The engine’s flash-fatbin router and the cache sizing below MUST agree — both read this one function.

kv_fp8_on

QWEN FP8-KV switch (MEMRA_KV_FP8 explicit; else the per-model KV_FP8_FORCE door set at model load; else OFF). Non-gemma full-attn layers hold e4m3 K/V via the kf8vf8 module.

latent_kv_bytes_per_token_for_plan

Context-linear bytes per token owned by StatePlan::LatentKvCache layers in [lo, hi), mirroring Cache::new_inner’s latent arm plus the engine’s lazy resident pool-key plane (lane/glm5-gpf-workspace, 2026-08-30).

set_swa_ring_default

swa_retain_from

The retain a ring-backed append must request: the aligned window start for first_row, minus the rewind slack, but NEVER below what the ring still holds.

swa_ring_on

swa_ring_rows

Physical rows required by the Step35 SWA reader contract. Prime starts at (base_len - (window - 1)) & !31, so at most 31 masked rows precede the live window.

tp_kv_rank_allocation_shape

wkv_on

FP8-WINDOWED switch (MEMRA_GEMMA_WKV; serving-mode default): SPEC serving (MEMRA_DRAFT set) -> OFF, plain -> ON — the acceptance-vs-depth record lives on the engine-side history of Engine::wkv_on (git). Explicit env always wins.