pub struct Dsv4State {Show 20 fields
pub window: Vec<Vec<f32>>,
pub compressed: Vec<Vec<f32>>,
pub index_kv: Vec<Vec<f32>>,
pub pending_kv: Vec<Vec<f32>>,
pub pending_score: Vec<Vec<f32>>,
pub prev_kv: Vec<Vec<f32>>,
pub prev_score: Vec<Vec<f32>>,
pub pending_ix_kv: Vec<Vec<f32>>,
pub pending_ix_score: Vec<Vec<f32>>,
pub prev_ix_kv: Vec<Vec<f32>>,
pub prev_ix_score: Vec<Vec<f32>>,
pub pos: usize,
pub kv_id: u64,
pub dev_filled: Vec<usize>,
pub dev_n_comp: Vec<usize>,
pub dev_n_ix: Vec<usize>,
pub dev_owned: bool,
pub dev_set: Vec<bool>,
pub partial_set: Vec<bool>,
pub split_deep: bool,
}Expand description
Per-sequence state. The compressor and the indexer each keep their own compressed cache and a partial window, so decode picks up mid-window exactly where prefill left off.
Fields§
§window: Vec<Vec<f32>>Sliding-window KV per layer, [window, head_dim] ring.
compressed: Vec<Vec<f32>>Compressed KV per layer, appended once per ratio tokens.
index_kv: Vec<Vec<f32>>The indexer’s own compressed cache per layer.
pending_kv: Vec<Vec<f32>>Partial window being accumulated, per layer: kv and score streams.
pending_score: Vec<Vec<f32>>§prev_kv: Vec<Vec<f32>>The window before it, kept only by the overlapping compressor — its fold reads half its dimensions from the previous stride.
prev_score: Vec<Vec<f32>>§pending_ix_kv: Vec<Vec<f32>>The indexer’s compressor runs alongside the attention one and keeps its own window — same shape, different width and different weights.
pending_ix_score: Vec<Vec<f32>>§prev_ix_kv: Vec<Vec<f32>>§prev_ix_score: Vec<Vec<f32>>§pos: usize§kv_id: u64Identifies this sequence’s caches on the device. A fresh state gets a fresh id, so a device buffer left over from the previous conversation can never be read as if it belonged to this one.
dev_filled: Vec<usize>When the token graph owns a layer’s caches, the CONTENTS live on the card and only these counts stay here — how much of the window is filled, and how many compressed entries each cache holds. All three follow from the position, so keeping them costs nothing and reading them back would cost a round trip.
dev_n_comp: Vec<usize>§dev_n_ix: Vec<usize>§dev_owned: boolTrue once this sequence has run a layer on the card with the device owning its state. The host copies above are stale from then on, so the CPU path must not be used for that layer again.
dev_set: Vec<bool>The device-layer set of the FIRST chained token. If it ever differs, some layer’s caches are on the wrong side and the answer would be quietly wrong — the loop refuses instead.
partial_set: Vec<bool>Which layers run their MoE on the card from a PARTIAL expert pack.
Their walk attention must stay on the host: the device attention
frame and the device MoE frame of one layer share pooled slots and
poison each other across tokens (see attention_step).
split_deep: boolMore than one layer walks past the device prefix. The stale-slot poison needs a CHAIN of walk frames handing state through the pooled slots; a single tail layer (the canonical shape) never chains and its device attention is measured exact.