Enum QTensor 

pub enum QTensor {
    F32 {
        data: Vec<f32>,
        rows: usize,
        cols: usize,
    },
    Mapped {
        model: Arc<CmfModel>,
        idx: usize,
        dtype: TensorDtype,
        rows: usize,
        cols: usize,
        row_scale: Vec<f32>,
        col_field: Vec<f32>,
        vbit_offsets: Vec<usize>,
        repack: Vec<u8>,
    },
}

Variants

F32

Fields

data: Vec<f32>

rows: usize

cols: usize

Mapped

Fields

model: Arc<CmfModel>

idx: usize

Index into the model’s tensor directory.

dtype: TensorDtype

rows: usize

cols: usize

row_scale: Vec<f32>

Per-row scales, dequantized to f32 up front (tiny).

col_field: Vec<f32>

q8_2f column field (θ), dequantized up front; empty for q8_row.

vbit_offsets: Vec<usize>

Vbit only: byte offset of each row’s packed data within the tensor blob ([rows + 1], computed once at load — the per- matvec prefix scan over row bit-widths was O(rows) each call).

repack: Vec<u8>

q8-family decode repack (load-time, optional): rows in groups of 4, interleaved in 16-byte units — one 64-byte line per iteration feeds all 4 sdot lanes, ONE sequential weight stream per worker instead of four (this is where llama.cpp’s repacked Q8 kernels get their bandwidth). Empty = off (CMF_REPACK=0, non-SDOT arch, or an ineligible shape). Trades an anonymous copy of the quants for mmap pages that go cold.

Implementations

impl QTensor

pub fn from_f32(data: Vec<f32>, rows: usize, cols: usize) -> Self

pub fn from_model(model: &Arc<CmfModel>, name: &str) -> Result<Self, String>

Wrap a directory tensor without dequantizing the payload. Falls back to dequantized f32 for dtypes without a fused kernel.

Examples found in repository

examples/dsv41_engram_component.rs (line 171)

fn main() -> Result<(), Box<dyn Error>> {
    let mut args = std::env::args_os().skip(1);
    let dir = PathBuf::from(args.next().ok_or("missing raw fixture directory")?);
    let layer = dir
        .file_name()
        .and_then(|s| s.to_str())
        .and_then(|s| s.strip_prefix("raw-layer-"))
        .ok_or("fixture directory must be named raw-layer-N")?;
    let prefix = format!("model.layers.{layer}.engram");
    let cmf = dir.join("engram-component.cmf");
    write_fixture(&dir, &cmf)?;
    let model = Arc::new(CmfModel::open(&cmf)?);
    let embed = RawFp8Rows::from_model(
        &model,
        &format!("{prefix}.embed.weight"),
        &format!("{prefix}.embed.scale"),
    )?;
    let wkv = QTensor::from_model(&model, &format!("{prefix}.wkv.weight"))?;
    let q_weight = f32_tensor(&model, &format!("{prefix}.q_weight"))?;
    let k_weight = f32_tensor(&model, &format!("{prefix}.k_weight"))?;
    let engram = Dsv41Engram {
        embed,
        wkv,
        q_weight,
        k_weight,
    };
    let cfg = cfg();
    let meta: Value = serde_json::from_slice(&fs::read(dir.join("manifest.json"))?)?;
    let mut max_abs = 0.0f32;
    let mut sum_abs = 0.0f64;
    let mut count = 0usize;
    let mut cases = 0usize;
    for case in 0..3 {
        let prefix = format!("case{case}");
        let input_shape = meta[&format!("{prefix}.input")]["shape"]
            .as_array()
            .unwrap();
        let seq = input_shape[1].as_u64().unwrap() as usize;
        let input = read_bf16(&dir.join(format!("{prefix}_input.bin")))?;
        let expected = read_bf16(&dir.join(format!("{prefix}_output.bin")))?;
        let index_bytes = fs::read(dir.join(format!("{prefix}_indices.bin")))?;
        let mut indices = Vec::with_capacity(index_bytes.len() / 8);
        for b in index_bytes.chunks_exact(8) {
            indices.push(u64::from_le_bytes(b.try_into().unwrap()) as usize);
        }
        let token_mask: Vec<bool> = if case == 1 {
            vec![
                true, true, true, true, true, false, false, false, true, true, true, true, true,
                true, true, true, true, true, true, true,
            ]
        } else {
            vec![true; seq]
        };
        for token in 0..seq {
            let mut h = input[token * 4 * 5120..(token + 1) * 4 * 5120].to_vec();
            let hashes = &indices[token * 24..(token + 1) * 24];
            dsv41_apply_engram_for_test(&engram, &mut h, hashes, &cfg, token_mask[token]);
            for (&got, &want) in h
                .iter()
                .zip(&expected[token * 4 * 5120..(token + 1) * 4 * 5120])
            {
                let d = (got - want).abs();
                max_abs = max_abs.max(d);
                sum_abs += d as f64;
                count += 1;
            }
        }
        cases += 1;
        println!("case={case} seq={seq} cumulative_max_abs={max_abs:.8e}",);
    }
    println!(
        "summary cases={cases} values={count} max_abs={max_abs:.8e} mean_abs={:.8e}",
        sum_abs / count.max(1) as f64
    );
    Ok(())
}

examples/matvec_bw.rs (line 55)

fn main() {
    let mut args = std::env::args().skip(1);
    let path = args
        .next()
        .expect("usage: matvec_bw <model.cmf> [sweep|one <tensor>]");
    let mode = args.next().unwrap_or_else(|| "sweep".to_string());
    let model = Arc::new(cortiq_core::CmfModel::open(&path).expect("open model"));

    // Real LM activations carry a few heavy channels (>8·rms); measured
    // mean on this model is ~3.7. NOUT models that distribution.
    let nout: usize = std::env::var("NOUT")
        .ok()
        .and_then(|v| v.parse().ok())
        .unwrap_or(4);
    let mk_x = |cols: usize| -> Vec<f32> {
        let mut x: Vec<f32> = (0..cols).map(|i| ((i % 17) as f32 - 8.0) / 8.0).collect();
        for k in 0..nout {
            x[k * 37 % cols] = 40.0;
        }
        x
    };

    if mode == "one" {
        let name = args
            .next()
            .unwrap_or_else(|| "model.embed_tokens.weight".to_string());
        let entry = model.tensor(&name).expect("tensor not found");
        let (rows, cols) = (entry.shape[0], entry.shape[1]);
        let nbytes = entry.nbytes as f64;
        println!(
            "tensor {name}: {rows}x{cols} {:?} = {:.1} MB, NOUT={nout}",
            entry.dtype,
            nbytes / 1e6
        );
        let t = QTensor::from_model(&model, &name).expect("wrap");
        let x = mk_x(cols);
        let mut out = vec![0f32; rows];
        t.matvec(&x, &mut out, None);
        for nt in [1usize, 2, 4, 6, 8, 10] {
            let pool = if nt == 1 { None } else { Some(Pool::new(nt)) };
            let iters = 8;
            t.matvec(&x, &mut out, pool.as_ref());
            let t0 = Instant::now();
            for _ in 0..iters {
                t.matvec(&x, &mut out, pool.as_ref());
            }
            let el = t0.elapsed().as_secs_f64();
            println!(
                "threads={nt:2}  {:6.2} ms/matvec  {:6.1} GB/s (sink {:.3})",
                el / iters as f64 * 1e3,
                nbytes * iters as f64 / el / 1e9,
                out[0]
            );
        }
        return;
    }

    // sweep: every 2-D q8_2f tensor once = one decode's worth of weights.
    let names: Vec<String> = model
        .tensors
        .iter()
        .filter(|t| t.dtype == TensorDtype::Q8_2f && t.shape.len() == 2)
        .map(|t| t.name.clone())
        .collect();
    let total_bytes: f64 = model
        .tensors
        .iter()
        .filter(|t| t.dtype == TensorDtype::Q8_2f && t.shape.len() == 2)
        .map(|t| t.nbytes as f64)
        .sum();
    println!(
        "sweep: {} q8_2f tensors, {:.2} GB total (= weights streamed per decode token), NOUT={nout}",
        names.len(),
        total_bytes / 1e9
    );

    let tensors: Vec<(QTensor, Vec<f32>, Vec<f32>)> = names
        .iter()
        .map(|n| {
            let e = model.tensor(n).unwrap();
            let (rows, cols) = (e.shape[0], e.shape[1]);
            (
                QTensor::from_model(&model, n).expect("wrap"),
                mk_x(cols),
                vec![0f32; rows],
            )
        })
        .collect();
    let mut tensors = tensors;

    // Whole-model residency pass BEFORE any timing: the first touch of a
    // 4.2 GB mmap faults ~260k pages, which would otherwise be charged to
    // whichever thread count happens to run first. REVERSE=1 flips the
    // order as a check that no first-touch cost is left in the table.
    for _ in 0..2 {
        for (t, x, out) in tensors.iter_mut() {
            t.matvec(x, out, None);
        }
    }
    let mut counts = vec![1usize, 2, 4, 6, 8, 10];
    if std::env::var("REVERSE").is_ok() {
        counts.reverse();
    }
    for nt in counts {
        let pool = if nt == 1 { None } else { Some(Pool::new(nt)) };
        // one warm pass, then two measured
        for (t, x, out) in tensors.iter_mut() {
            t.matvec(x, out, pool.as_ref());
        }
        let iters = 2;
        let t0 = Instant::now();
        for _ in 0..iters {
            for (t, x, out) in tensors.iter_mut() {
                t.matvec(x, out, pool.as_ref());
            }
        }
        let el = t0.elapsed().as_secs_f64();
        let per_tok = el / iters as f64;
        println!(
            "threads={nt:2}  {:7.1} ms/sweep  {:6.1} GB/s  -> weight-path-only ceiling {:5.1} tok/s",
            per_tok * 1e3,
            total_bytes * iters as f64 / el / 1e9,
            1.0 / per_tok
        );
    }
}

examples/dsv41_engram_proof.rs (line 353)

fn verify_component_layer(
    root: &Path,
    projected_root: &Path,
    layer: usize,
    reference_hashes: &[Vec<Vec<Vec<usize>>>],
) -> Result<(), Box<dyn Error>> {
    let dir = root.join(format!("raw-layer-{layer}"));
    let layer_meta: Value =
        serde_json::from_slice(&fs::read(root.join(format!("layer-{layer}.json")))?)?;
    let original_rows: Vec<usize> = serde_json::from_value(layer_meta["original_row_ids"].clone())?;
    let row_to_compact: HashMap<usize, usize> = original_rows
        .iter()
        .copied()
        .enumerate()
        .map(|(compact, original)| (original, compact))
        .collect();
    let prefix = format!("model.layers.{layer}.engram");
    let cmf = dir.join("engram-component.cmf");
    let model = Arc::new(CmfModel::open(&cmf)?);
    let embed = RawFp8Rows::from_model(
        &model,
        &format!("{prefix}.embed.weight"),
        &format!("{prefix}.embed.scale"),
    )?;
    let wkv = QTensor::from_model(&model, &format!("{prefix}.wkv.weight"))?;
    let q_weight = f32_tensor(&model, &format!("{prefix}.q_weight"))?;
    let k_weight = f32_tensor(&model, &format!("{prefix}.k_weight"))?;
    let engram = Dsv41Engram {
        embed,
        wkv,
        q_weight,
        k_weight,
    };
    let cases = layer_meta["cases"]
        .as_array()
        .ok_or("layer cases missing")?;
    for (case_no, case) in cases.iter().enumerate() {
        let original_hashes: Vec<Vec<usize>> =
            serde_json::from_value(case["original_hash_ids"].clone())?;
        if original_hashes.len() != reference_hashes[case_no].len() {
            return Err(format!("layer {layer} case {case_no} sequence length mismatch").into());
        }
        let mut expected_compact = Vec::with_capacity(original_hashes.len() * HASH_COLS);
        for (pos, rows) in original_hashes.iter().enumerate() {
            if rows.len() != HASH_COLS || reference_hashes[case_no][pos][0].len() != HASH_COLS {
                return Err(format!("layer {layer} case {case_no} hash width mismatch").into());
            }
            let layer_index = if layer == 1 { 0 } else { 1 };
            if rows != &reference_hashes[case_no][pos][layer_index] {
                return Err(format!(
                    "layer {layer} case {case_no} original indices differ from full hash oracle at position {pos}"
                )
                .into());
            }
            for &row in rows {
                expected_compact.push(
                    *row_to_compact
                        .get(&row)
                        .ok_or_else(|| format!("layer {layer} missing compact row for {row}"))?,
                );
            }
        }
        let got_compact = read_i64(&dir.join(format!("case{case_no}_indices.bin")))?;
        if got_compact != expected_compact {
            let first = got_compact
                .iter()
                .zip(&expected_compact)
                .enumerate()
                .find(|(_, (a, b))| a != b);
            return Err(format!(
                "layer {layer} case {case_no} compact indices mismatch: {first:?}"
            )
            .into());
        }

        let seq = original_hashes.len();
        let mut gathered = vec![0.0f32; seq * HASH_COLS * 256];
        for token in 0..seq {
            for col in 0..HASH_COLS {
                engram.embed.row_into(
                    got_compact[token * HASH_COLS + col],
                    &mut gathered
                        [(token * HASH_COLS + col) * 256..(token * HASH_COLS + col + 1) * 256],
                );
            }
        }
        gathered.iter_mut().for_each(|v| *v = bf16_roundtrip(*v));
        compare_bf16(
            &format!("layer={layer} case={case_no} gathered"),
            &gathered,
            &read_bf16(&dir.join(format!("case{case_no}_gathered.bin")))?,
        )?;

        let mut projected = Vec::with_capacity(seq * (DIM * (HC_MULT + 1)));
        for token in 0..seq {
            let input = &gathered[token * HASH_COLS * 256..(token + 1) * HASH_COLS * 256];
            let mut row = vec![0.0f32; DIM * (HC_MULT + 1)];
            engram.wkv.matvec(input, &mut row, None);
            row.iter_mut().for_each(|v| *v = bf16_roundtrip(*v));
            projected.extend_from_slice(&row);
        }
        compare_bf16(
            &format!("layer={layer} case={case_no} projected"),
            &projected,
            &read_bf16(&projected_root.join(format!("layer-{layer}/case{case_no}_projected.bin")))?,
        )?;

        let input = read_bf16(&dir.join(format!("case{case_no}_input.bin")))?;
        let expected_output = read_bf16(&dir.join(format!("case{case_no}_output.bin")))?;
        let mask: Option<Vec<bool>> = if case["token_mask"].is_null() {
            None
        } else {
            Some(serde_json::from_value(case["token_mask"].clone())?)
        };
        let mut output = Vec::with_capacity(input.len());
        for token in 0..seq {
            let mut h = input[token * HC_MULT * DIM..(token + 1) * HC_MULT * DIM].to_vec();
            dsv41_apply_engram_for_test(
                &engram,
                &mut h,
                &got_compact[token * HASH_COLS..(token + 1) * HASH_COLS],
                &cfg(),
                mask.as_ref().map(|m| m[token]).unwrap_or(true),
            );
            output.extend_from_slice(&h);
        }
        compare_bf16(
            &format!("layer={layer} case={case_no} output"),
            &output,
            &expected_output,
        )?;
        println!(
            "component layer={} case={} name={} compact_indices=true gathered=true projected=true output=true",
            layer,
            case_no,
            case["name"].as_str().unwrap_or("?")
        );
    }
    Ok(())
}

pub fn model_dtype(&self) -> Option<TensorDtype>

The layout this tensor is stored in, when it is mapped from a model. The frames branch on it — a q2tp gate against a q4tp down is a real combination in the 2-bit profile and needs a different kernel.

pub fn model_idx(&self) -> Option<usize>

The tensor’s index in the model directory, when it is mapped from one. The GPU frames bind by index rather than by name — a name lookup per layer per token is not free, and the index is what the device cache is keyed on anyway.

pub fn model_arc(&self) -> Option<Arc<CmfModel>>

The model this tensor is mapped from, when it is mapped at all. The GPU frames need the container to reach the bytes; a QTensor already holds it, and threading a second handle down every call site to say the same thing invites the two to disagree.

pub fn rows(&self) -> usize

pub fn mapped_q4tp(&self) -> Option<(&Arc<CmfModel>, usize)>

Same slot as mapped_q4t for a q4tp tensor — the fused DiT FFN picks its kernels by which of the two answers.

pub fn mapped_device_gemm(&self) -> Option<(&Arc<CmfModel>, usize)>

(model, tensor idx) for a mapped weight in ANY codec the fused device paths can run — four-bit tiled or either int8 layout.

The fused DiT chains asked for mapped_q4tp by name, so an eight-bit container never reached them and rendered through per-op GEMMs even after those kernels learned its codec. The gate is what the codec has a device GEMM for, not which codec it is.

pub fn mapped_q2tp(&self) -> Option<(&Arc<CmfModel>, usize)>

(model, tensor idx) for a q2tp mapped weight — the 2-bit twin of mapped_q4tp, used by the mixed MoE profile.

pub fn cols(&self) -> usize

pub fn mapped_q1(&self) -> Option<(&Arc<CmfModel>, usize)>

(model, tensor idx) for a q1 mapped weight — the wgpu token graph keys its resident VRAM cache by idx. None for any other dtype/kind.

pub fn graph_weight(&self) -> Option<(&Arc<CmfModel>, usize, u8, &[f32])>

(model, idx, kind, row_scale) for a graph-capable mapped weight. kind: 0=q8_row (per-row scales), 1=q1, 2=q4_block, 3=q1t (tile-embedded, no rs), 5=q4_tiled, 6=q4tp, 7=q8_2f (both scale planes live inside the tensor). None only for vbit.

The old comment here claimed q4_block was unhandled while the arm right below mapped it, and it named q8_2f as unhandled after that stopped being true — a stale comment on this function is how a model silently loses the graph, so it is worth keeping honest.

pub fn as_f32(&self) -> Option<&[f32]>

Dense f32 view — only for owned tensors. Masked/sparse execution paths require it; quantized weights don’t support masks yet.

pub fn row_f32(&self, r: usize, dst: &mut [f32])

Dequantize one row into dst (embedding lookup).

pub fn sparse_col_ok(&self) -> bool

Can this tensor’s columns be read cheaply (for sparse down_proj)? True for F32/Q8Row/Q8_2f (per-row scale, direct strided access); false for group-packed q4/vbit (column access would unpack whole groups — sparse execution falls back to f32 for those).

pub fn add_col_scaled(&self, c: usize, w: f32, out: &mut [f32])

down_proj [hidden, inter]: accumulate w · col(c) into out [hidden] — reads ONLY column c (one neuron) from the mmap, no full-matrix dequant. out[k] += w · down[k, c].

pub fn prefetch_row(&self, r: usize)

Touch the head of row r so the DRAM latency of the next neuron’s weights overlaps the current one’s arithmetic.

Scattered rows are what per-token sparsity reads, and a 2 KB stride is past what the hardware prefetcher follows: without this every row starts with a cold miss that nothing hides. One touch per 512 bytes is enough — the rest of the row is a sequential run the prefetcher does pick up.

pub fn add_row_scaled( &self, r: usize, w: f32, out: &mut [f32], scratch: &mut [f32], )

out += w · row(r) — the transposed twin of add_col_scaled.

A neuron’s down weights are a COLUMN of [hidden, inter], and a column is strided: reading one costs a cache line per element, so per-neuron dynamic sparsity saves arithmetic and no bytes. Stored transposed (down_proj.t.weight, [inter, hidden]) the same weights are a contiguous ROW, and this accumulate reads exactly the neurons the token asked for.

pub fn row_dot(&self, r: usize, x: &[f32], scratch: &mut [f32]) -> f32

Dot of row r with x (gate/up active-neuron path). Reads only row r from the mmap — no full dequant. q4/vbit dequant the row into scratch first (rare for active-FFN weights).

pub fn matvec(&self, x: &[f32], out: &mut [f32], pool: Option<&Pool>)

out = W · x (row-major). F32 delegates to the historical bit-exact path; Mapped runs the fused int8 kernel.

Examples found in repository

examples/matvec_bw.rs (line 58)

fn main() {
    let mut args = std::env::args().skip(1);
    let path = args
        .next()
        .expect("usage: matvec_bw <model.cmf> [sweep|one <tensor>]");
    let mode = args.next().unwrap_or_else(|| "sweep".to_string());
    let model = Arc::new(cortiq_core::CmfModel::open(&path).expect("open model"));

    // Real LM activations carry a few heavy channels (>8·rms); measured
    // mean on this model is ~3.7. NOUT models that distribution.
    let nout: usize = std::env::var("NOUT")
        .ok()
        .and_then(|v| v.parse().ok())
        .unwrap_or(4);
    let mk_x = |cols: usize| -> Vec<f32> {
        let mut x: Vec<f32> = (0..cols).map(|i| ((i % 17) as f32 - 8.0) / 8.0).collect();
        for k in 0..nout {
            x[k * 37 % cols] = 40.0;
        }
        x
    };

    if mode == "one" {
        let name = args
            .next()
            .unwrap_or_else(|| "model.embed_tokens.weight".to_string());
        let entry = model.tensor(&name).expect("tensor not found");
        let (rows, cols) = (entry.shape[0], entry.shape[1]);
        let nbytes = entry.nbytes as f64;
        println!(
            "tensor {name}: {rows}x{cols} {:?} = {:.1} MB, NOUT={nout}",
            entry.dtype,
            nbytes / 1e6
        );
        let t = QTensor::from_model(&model, &name).expect("wrap");
        let x = mk_x(cols);
        let mut out = vec![0f32; rows];
        t.matvec(&x, &mut out, None);
        for nt in [1usize, 2, 4, 6, 8, 10] {
            let pool = if nt == 1 { None } else { Some(Pool::new(nt)) };
            let iters = 8;
            t.matvec(&x, &mut out, pool.as_ref());
            let t0 = Instant::now();
            for _ in 0..iters {
                t.matvec(&x, &mut out, pool.as_ref());
            }
            let el = t0.elapsed().as_secs_f64();
            println!(
                "threads={nt:2}  {:6.2} ms/matvec  {:6.1} GB/s (sink {:.3})",
                el / iters as f64 * 1e3,
                nbytes * iters as f64 / el / 1e9,
                out[0]
            );
        }
        return;
    }

    // sweep: every 2-D q8_2f tensor once = one decode's worth of weights.
    let names: Vec<String> = model
        .tensors
        .iter()
        .filter(|t| t.dtype == TensorDtype::Q8_2f && t.shape.len() == 2)
        .map(|t| t.name.clone())
        .collect();
    let total_bytes: f64 = model
        .tensors
        .iter()
        .filter(|t| t.dtype == TensorDtype::Q8_2f && t.shape.len() == 2)
        .map(|t| t.nbytes as f64)
        .sum();
    println!(
        "sweep: {} q8_2f tensors, {:.2} GB total (= weights streamed per decode token), NOUT={nout}",
        names.len(),
        total_bytes / 1e9
    );

    let tensors: Vec<(QTensor, Vec<f32>, Vec<f32>)> = names
        .iter()
        .map(|n| {
            let e = model.tensor(n).unwrap();
            let (rows, cols) = (e.shape[0], e.shape[1]);
            (
                QTensor::from_model(&model, n).expect("wrap"),
                mk_x(cols),
                vec![0f32; rows],
            )
        })
        .collect();
    let mut tensors = tensors;

    // Whole-model residency pass BEFORE any timing: the first touch of a
    // 4.2 GB mmap faults ~260k pages, which would otherwise be charged to
    // whichever thread count happens to run first. REVERSE=1 flips the
    // order as a check that no first-touch cost is left in the table.
    for _ in 0..2 {
        for (t, x, out) in tensors.iter_mut() {
            t.matvec(x, out, None);
        }
    }
    let mut counts = vec![1usize, 2, 4, 6, 8, 10];
    if std::env::var("REVERSE").is_ok() {
        counts.reverse();
    }
    for nt in counts {
        let pool = if nt == 1 { None } else { Some(Pool::new(nt)) };
        // one warm pass, then two measured
        for (t, x, out) in tensors.iter_mut() {
            t.matvec(x, out, pool.as_ref());
        }
        let iters = 2;
        let t0 = Instant::now();
        for _ in 0..iters {
            for (t, x, out) in tensors.iter_mut() {
                t.matvec(x, out, pool.as_ref());
            }
        }
        let el = t0.elapsed().as_secs_f64();
        let per_tok = el / iters as f64;
        println!(
            "threads={nt:2}  {:7.1} ms/sweep  {:6.1} GB/s  -> weight-path-only ceiling {:5.1} tok/s",
            per_tok * 1e3,
            total_bytes * iters as f64 / el / 1e9,
            1.0 / per_tok
        );
    }
}

examples/dsv41_engram_proof.rs (line 426)

fn verify_component_layer(
    root: &Path,
    projected_root: &Path,
    layer: usize,
    reference_hashes: &[Vec<Vec<Vec<usize>>>],
) -> Result<(), Box<dyn Error>> {
    let dir = root.join(format!("raw-layer-{layer}"));
    let layer_meta: Value =
        serde_json::from_slice(&fs::read(root.join(format!("layer-{layer}.json")))?)?;
    let original_rows: Vec<usize> = serde_json::from_value(layer_meta["original_row_ids"].clone())?;
    let row_to_compact: HashMap<usize, usize> = original_rows
        .iter()
        .copied()
        .enumerate()
        .map(|(compact, original)| (original, compact))
        .collect();
    let prefix = format!("model.layers.{layer}.engram");
    let cmf = dir.join("engram-component.cmf");
    let model = Arc::new(CmfModel::open(&cmf)?);
    let embed = RawFp8Rows::from_model(
        &model,
        &format!("{prefix}.embed.weight"),
        &format!("{prefix}.embed.scale"),
    )?;
    let wkv = QTensor::from_model(&model, &format!("{prefix}.wkv.weight"))?;
    let q_weight = f32_tensor(&model, &format!("{prefix}.q_weight"))?;
    let k_weight = f32_tensor(&model, &format!("{prefix}.k_weight"))?;
    let engram = Dsv41Engram {
        embed,
        wkv,
        q_weight,
        k_weight,
    };
    let cases = layer_meta["cases"]
        .as_array()
        .ok_or("layer cases missing")?;
    for (case_no, case) in cases.iter().enumerate() {
        let original_hashes: Vec<Vec<usize>> =
            serde_json::from_value(case["original_hash_ids"].clone())?;
        if original_hashes.len() != reference_hashes[case_no].len() {
            return Err(format!("layer {layer} case {case_no} sequence length mismatch").into());
        }
        let mut expected_compact = Vec::with_capacity(original_hashes.len() * HASH_COLS);
        for (pos, rows) in original_hashes.iter().enumerate() {
            if rows.len() != HASH_COLS || reference_hashes[case_no][pos][0].len() != HASH_COLS {
                return Err(format!("layer {layer} case {case_no} hash width mismatch").into());
            }
            let layer_index = if layer == 1 { 0 } else { 1 };
            if rows != &reference_hashes[case_no][pos][layer_index] {
                return Err(format!(
                    "layer {layer} case {case_no} original indices differ from full hash oracle at position {pos}"
                )
                .into());
            }
            for &row in rows {
                expected_compact.push(
                    *row_to_compact
                        .get(&row)
                        .ok_or_else(|| format!("layer {layer} missing compact row for {row}"))?,
                );
            }
        }
        let got_compact = read_i64(&dir.join(format!("case{case_no}_indices.bin")))?;
        if got_compact != expected_compact {
            let first = got_compact
                .iter()
                .zip(&expected_compact)
                .enumerate()
                .find(|(_, (a, b))| a != b);
            return Err(format!(
                "layer {layer} case {case_no} compact indices mismatch: {first:?}"
            )
            .into());
        }

        let seq = original_hashes.len();
        let mut gathered = vec![0.0f32; seq * HASH_COLS * 256];
        for token in 0..seq {
            for col in 0..HASH_COLS {
                engram.embed.row_into(
                    got_compact[token * HASH_COLS + col],
                    &mut gathered
                        [(token * HASH_COLS + col) * 256..(token * HASH_COLS + col + 1) * 256],
                );
            }
        }
        gathered.iter_mut().for_each(|v| *v = bf16_roundtrip(*v));
        compare_bf16(
            &format!("layer={layer} case={case_no} gathered"),
            &gathered,
            &read_bf16(&dir.join(format!("case{case_no}_gathered.bin")))?,
        )?;

        let mut projected = Vec::with_capacity(seq * (DIM * (HC_MULT + 1)));
        for token in 0..seq {
            let input = &gathered[token * HASH_COLS * 256..(token + 1) * HASH_COLS * 256];
            let mut row = vec![0.0f32; DIM * (HC_MULT + 1)];
            engram.wkv.matvec(input, &mut row, None);
            row.iter_mut().for_each(|v| *v = bf16_roundtrip(*v));
            projected.extend_from_slice(&row);
        }
        compare_bf16(
            &format!("layer={layer} case={case_no} projected"),
            &projected,
            &read_bf16(&projected_root.join(format!("layer-{layer}/case{case_no}_projected.bin")))?,
        )?;

        let input = read_bf16(&dir.join(format!("case{case_no}_input.bin")))?;
        let expected_output = read_bf16(&dir.join(format!("case{case_no}_output.bin")))?;
        let mask: Option<Vec<bool>> = if case["token_mask"].is_null() {
            None
        } else {
            Some(serde_json::from_value(case["token_mask"].clone())?)
        };
        let mut output = Vec::with_capacity(input.len());
        for token in 0..seq {
            let mut h = input[token * HC_MULT * DIM..(token + 1) * HC_MULT * DIM].to_vec();
            dsv41_apply_engram_for_test(
                &engram,
                &mut h,
                &got_compact[token * HASH_COLS..(token + 1) * HASH_COLS],
                &cfg(),
                mask.as_ref().map(|m| m[token]).unwrap_or(true),
            );
            output.extend_from_slice(&h);
        }
        compare_bf16(
            &format!("layer={layer} case={case_no} output"),
            &output,
            &expected_output,
        )?;
        println!(
            "component layer={} case={} name={} compact_indices=true gathered=true projected=true output=true",
            layer,
            case_no,
            case["name"].as_str().unwrap_or("?")
        );
    }
    Ok(())
}

pub fn matvec2( &self, x1: &[f32], x2: &[f32], o1: &mut [f32], o2: &mut [f32], pool: Option<&Pool>, )

Fused two-input matvec (MTP verify pair): weights streamed once.

impl QTensor

pub fn q4tp_mapped(&self) -> Option<(&Arc<CmfModel>, usize)>

Batched matvec (prefill-GEMM): xs — row-major [b, cols], out — row-major [b, rows]. Element-wise semantics are IDENTICAL to b matvec calls (same dot kernels in the same order); the win — the weight row streams from DRAM once per batch, not b times. (model, index) when this is a memory-mapped q4tp tensor — the identity a device-resident chain needs to hand tp_matmat the weight without going through this struct’s own dispatch.

pub fn matmat( &self, xs_all: &[f32], b: usize, out: &mut [f32], pool: Option<&Pool>, )

impl QTensor

pub fn device_matmat(&self, xs: &[f32], b: usize, out: &mut [f32]) -> bool

The device GEMM this tensor would take, run once on the caller’s data — the startup parity probe’s arm, and the one place that knows which entry point each codec has.

It exists because the probe used to look for a q4tp weight by name AND dtype, and a container packed any other way was declared “host path” for the whole render even though its codec had a device GEMM of its own. A gate that only recognizes one codec is a gate that silently downgrades every other one.

pub fn matvec_many<const N: usize>( ts: [&QTensor; N], x: &[f32], outs: [&mut [f32]; N], pool: Option<&Pool>, )

Multi-matrix job (roadmap §3 P0): N tensors sharing one input run under a SINGLE pool dispatch — QKV or gate+up cost one barrier instead of N. Per-row math is the exact same kernel as matvec (bit-identical outputs); only the dispatch is fused. Falls back to N sequential matvecs when the set is not a uniform q8-family/F32 group or there is no pool.

impl QTensor

pub fn matvec2_many<const N: usize>( ts: [&QTensor; N], x1: &[f32], x2: &[f32], o1s: [&mut [f32]; N], o2s: [&mut [f32]; N], pool: Option<&Pool>, )

Pair-input multi-matrix job: N tensors × 2 shared inputs under a single pool dispatch — the MTP/pair decode path publishes one job for Q/K/V (and one for gate+up) instead of one per tensor. Per-row math is exactly matvec2’s kernels; bit-identical.

pub fn matvec_silu_mul( gate: &QTensor, up: &QTensor, x: &[f32], out: &mut [f32], pool: Option<&Pool>, ) -> bool

Fused gate+up matvec with SiLU·mul: for each row r, computes silu(gate·x) * (up·x) and writes to out[r]. ONE pool dispatch, no intermediate g/u buffers, no separate silu pass. Falls back (returns false) for unsupported dtype combos.

pub fn moe_gate_up_many( pairs: &[(&QTensor, &QTensor)], x: &[f32], outs: &mut [Vec<f32>], pool: Option<&Pool>, ) -> bool

Every routed expert’s fused gate/up/SiLU under ONE pool dispatch.

The per-expert path pays a pool barrier per expert per stage: at 9 experts over 40 layers that is ~720 barriers a token, and a decode profile of Qwen3.6-35B-A3B showed the pool parked in psynch_cvwait about twice as long as it spent computing. Laying every expert’s rows end-to-end in one virtual row space collapses the stage to a single dispatch. The per-row body is the single-expert q4tp arm verbatim, so outputs are bit-identical.

false = something is outside the fused q4tp kernel (dtype, shape, or the CMF_SDOT=0 exact contract); the caller walks the ordinary per-expert path.

pub fn moe_down_many( downs: &[&QTensor], gs: &[Vec<f32>], weights: &[f32], out: &mut [f32], pool: Option<&Pool>, ) -> bool

Every routed expert’s down projection, weighted and summed into out, under ONE pool dispatch.

Partitioned by OUTPUT row rather than by expert: each row is owned by a single worker, so the experts are summed in the caller’s order — the same sequence of f32 adds the serial out[i] += w·eo[i] loop performs, hence bit-identical. Partitioning by expert instead would race on the shared accumulator.