Enum QTensor 

pub enum QTensor {
    F32 {
        data: Vec<f32>,
        rows: usize,
        cols: usize,
    },
    Mapped {
        model: Arc<CmfModel>,
        idx: usize,
        dtype: TensorDtype,
        rows: usize,
        cols: usize,
        row_scale: Vec<f32>,
        col_field: Vec<f32>,
        vbit_offsets: Vec<usize>,
        repack: Vec<u8>,
    },
}

Variants

F32

Fields

data: Vec<f32>

rows: usize

cols: usize

Mapped

Fields

model: Arc<CmfModel>

idx: usize

Index into the model’s tensor directory.

dtype: TensorDtype

rows: usize

cols: usize

row_scale: Vec<f32>

Per-row scales, dequantized to f32 up front (tiny).

col_field: Vec<f32>

q8_2f column field (θ), dequantized up front; empty for q8_row.

vbit_offsets: Vec<usize>

Vbit only: byte offset of each row’s packed data within the tensor blob ([rows + 1], computed once at load — the per- matvec prefix scan over row bit-widths was O(rows) each call).

repack: Vec<u8>

q8-family decode repack (load-time, optional): rows in groups of 4, interleaved in 16-byte units — one 64-byte line per iteration feeds all 4 sdot lanes, ONE sequential weight stream per worker instead of four (this is where llama.cpp’s repacked Q8 kernels get their bandwidth). Empty = off (CMF_REPACK=0, non-SDOT arch, or an ineligible shape). Trades an anonymous copy of the quants for mmap pages that go cold.

Implementations

impl QTensor

pub fn from_f32(data: Vec<f32>, rows: usize, cols: usize) -> Self

pub fn from_model(model: &Arc<CmfModel>, name: &str) -> Result<Self, String>

Wrap a directory tensor without dequantizing the payload. Falls back to dequantized f32 for dtypes without a fused kernel.

Examples found in repository

examples/dsv41_engram_component.rs (line 171)

fn main() -> Result<(), Box<dyn Error>> {
    let mut args = std::env::args_os().skip(1);
    let dir = PathBuf::from(args.next().ok_or("missing raw fixture directory")?);
    let layer = dir
        .file_name()
        .and_then(|s| s.to_str())
        .and_then(|s| s.strip_prefix("raw-layer-"))
        .ok_or("fixture directory must be named raw-layer-N")?;
    let prefix = format!("model.layers.{layer}.engram");
    let cmf = dir.join("engram-component.cmf");
    write_fixture(&dir, &cmf)?;
    let model = Arc::new(CmfModel::open(&cmf)?);
    let embed = RawFp8Rows::from_model(
        &model,
        &format!("{prefix}.embed.weight"),
        &format!("{prefix}.embed.scale"),
    )?;
    let wkv = QTensor::from_model(&model, &format!("{prefix}.wkv.weight"))?;
    let q_weight = f32_tensor(&model, &format!("{prefix}.q_weight"))?;
    let k_weight = f32_tensor(&model, &format!("{prefix}.k_weight"))?;
    let engram = Dsv41Engram {
        embed,
        wkv,
        q_weight,
        k_weight,
    };
    let cfg = cfg();
    let meta: Value = serde_json::from_slice(&fs::read(dir.join("manifest.json"))?)?;
    let mut max_abs = 0.0f32;
    let mut sum_abs = 0.0f64;
    let mut count = 0usize;
    let mut cases = 0usize;
    for case in 0..3 {
        let prefix = format!("case{case}");
        let input_shape = meta[&format!("{prefix}.input")]["shape"]
            .as_array()
            .unwrap();
        let seq = input_shape[1].as_u64().unwrap() as usize;
        let input = read_bf16(&dir.join(format!("{prefix}_input.bin")))?;
        let expected = read_bf16(&dir.join(format!("{prefix}_output.bin")))?;
        let index_bytes = fs::read(dir.join(format!("{prefix}_indices.bin")))?;
        let mut indices = Vec::with_capacity(index_bytes.len() / 8);
        for b in index_bytes.chunks_exact(8) {
            indices.push(u64::from_le_bytes(b.try_into().unwrap()) as usize);
        }
        let token_mask: Vec<bool> = if case == 1 {
            vec![
                true, true, true, true, true, false, false, false, true, true, true, true, true,
                true, true, true, true, true, true, true,
            ]
        } else {
            vec![true; seq]
        };
        for token in 0..seq {
            let mut h = input[token * 4 * 5120..(token + 1) * 4 * 5120].to_vec();
            let hashes = &indices[token * 24..(token + 1) * 24];
            dsv41_apply_engram_for_test(&engram, &mut h, hashes, &cfg, token_mask[token]);
            for (&got, &want) in h
                .iter()
                .zip(&expected[token * 4 * 5120..(token + 1) * 4 * 5120])
            {
                let d = (got - want).abs();
                max_abs = max_abs.max(d);
                sum_abs += d as f64;
                count += 1;
            }
        }
        cases += 1;
        println!("case={case} seq={seq} cumulative_max_abs={max_abs:.8e}",);
    }
    println!(
        "summary cases={cases} values={count} max_abs={max_abs:.8e} mean_abs={:.8e}",
        sum_abs / count.max(1) as f64
    );
    Ok(())
}

examples/matvec_bw.rs (line 55)

fn main() {
    let mut args = std::env::args().skip(1);
    let path = args
        .next()
        .expect("usage: matvec_bw <model.cmf> [sweep|one <tensor>]");
    let mode = args.next().unwrap_or_else(|| "sweep".to_string());
    let model = Arc::new(cortiq_core::CmfModel::open(&path).expect("open model"));

    // Real LM activations carry a few heavy channels (>8·rms); measured
    // mean on this model is ~3.7. NOUT models that distribution.
    let nout: usize = std::env::var("NOUT")
        .ok()
        .and_then(|v| v.parse().ok())
        .unwrap_or(4);
    let mk_x = |cols: usize| -> Vec<f32> {
        let mut x: Vec<f32> = (0..cols).map(|i| ((i % 17) as f32 - 8.0) / 8.0).collect();
        for k in 0..nout {
            x[k * 37 % cols] = 40.0;
        }
        x
    };

    if mode == "one" {
        let name = args
            .next()
            .unwrap_or_else(|| "model.embed_tokens.weight".to_string());
        let entry = model.tensor(&name).expect("tensor not found");
        let (rows, cols) = (entry.shape[0], entry.shape[1]);
        let nbytes = entry.nbytes as f64;
        println!(
            "tensor {name}: {rows}x{cols} {:?} = {:.1} MB, NOUT={nout}",
            entry.dtype,
            nbytes / 1e6
        );
        let t = QTensor::from_model(&model, &name).expect("wrap");
        let x = mk_x(cols);
        let mut out = vec![0f32; rows];
        t.matvec(&x, &mut out, None);
        for nt in [1usize, 2, 4, 6, 8, 10] {
            let pool = if nt == 1 { None } else { Some(Pool::new(nt)) };
            let iters = 8;
            t.matvec(&x, &mut out, pool.as_ref());
            let t0 = Instant::now();
            for _ in 0..iters {
                t.matvec(&x, &mut out, pool.as_ref());
            }
            let el = t0.elapsed().as_secs_f64();
            println!(
                "threads={nt:2}  {:6.2} ms/matvec  {:6.1} GB/s (sink {:.3})",
                el / iters as f64 * 1e3,
                nbytes * iters as f64 / el / 1e9,
                out[0]
            );
        }
        return;
    }

    // sweep: every 2-D q8_2f tensor once = one decode's worth of weights.
    let names: Vec<String> = model
        .tensors
        .iter()
        .filter(|t| t.dtype == TensorDtype::Q8_2f && t.shape.len() == 2)
        .map(|t| t.name.clone())
        .collect();
    let total_bytes: f64 = model
        .tensors
        .iter()
        .filter(|t| t.dtype == TensorDtype::Q8_2f && t.shape.len() == 2)
        .map(|t| t.nbytes as f64)
        .sum();
    println!(
        "sweep: {} q8_2f tensors, {:.2} GB total (= weights streamed per decode token), NOUT={nout}",
        names.len(),
        total_bytes / 1e9
    );

    let tensors: Vec<(QTensor, Vec<f32>, Vec<f32>)> = names
        .iter()
        .map(|n| {
            let e = model.tensor(n).unwrap();
            let (rows, cols) = (e.shape[0], e.shape[1]);
            (
                QTensor::from_model(&model, n).expect("wrap"),
                mk_x(cols),
                vec![0f32; rows],
            )
        })
        .collect();
    let mut tensors = tensors;

    // Whole-model residency pass BEFORE any timing: the first touch of a
    // 4.2 GB mmap faults ~260k pages, which would otherwise be charged to
    // whichever thread count happens to run first. REVERSE=1 flips the
    // order as a check that no first-touch cost is left in the table.
    for _ in 0..2 {
        for (t, x, out) in tensors.iter_mut() {
            t.matvec(x, out, None);
        }
    }
    let mut counts = vec![1usize, 2, 4, 6, 8, 10];
    if std::env::var("REVERSE").is_ok() {
        counts.reverse();
    }
    for nt in counts {
        let pool = if nt == 1 { None } else { Some(Pool::new(nt)) };
        // one warm pass, then two measured
        for (t, x, out) in tensors.iter_mut() {
            t.matvec(x, out, pool.as_ref());
        }
        let iters = 2;
        let t0 = Instant::now();
        for _ in 0..iters {
            for (t, x, out) in tensors.iter_mut() {
                t.matvec(x, out, pool.as_ref());
            }
        }
        let el = t0.elapsed().as_secs_f64();
        let per_tok = el / iters as f64;
        println!(
            "threads={nt:2}  {:7.1} ms/sweep  {:6.1} GB/s  -> weight-path-only ceiling {:5.1} tok/s",
            per_tok * 1e3,
            total_bytes * iters as f64 / el / 1e9,
            1.0 / per_tok
        );
    }
}

examples/dsv41_engram_proof.rs (line 353)

fn verify_component_layer(
    root: &Path,
    projected_root: &Path,
    layer: usize,
    reference_hashes: &[Vec<Vec<Vec<usize>>>],
) -> Result<(), Box<dyn Error>> {
    let dir = root.join(format!("raw-layer-{layer}"));
    let layer_meta: Value =
        serde_json::from_slice(&fs::read(root.join(format!("layer-{layer}.json")))?)?;
    let original_rows: Vec<usize> = serde_json::from_value(layer_meta["original_row_ids"].clone())?;
    let row_to_compact: HashMap<usize, usize> = original_rows
        .iter()
        .copied()
        .enumerate()
        .map(|(compact, original)| (original, compact))
        .collect();
    let prefix = format!("model.layers.{layer}.engram");
    let cmf = dir.join("engram-component.cmf");
    let model = Arc::new(CmfModel::open(&cmf)?);
    let embed = RawFp8Rows::from_model(
        &model,
        &format!("{prefix}.embed.weight"),
        &format!("{prefix}.embed.scale"),
    )?;
    let wkv = QTensor::from_model(&model, &format!("{prefix}.wkv.weight"))?;
    let q_weight = f32_tensor(&model, &format!("{prefix}.q_weight"))?;
    let k_weight = f32_tensor(&model, &format!("{prefix}.k_weight"))?;
    let engram = Dsv41Engram {
        embed,
        wkv,
        q_weight,
        k_weight,
    };
    let cases = layer_meta["cases"]
        .as_array()
        .ok_or("layer cases missing")?;
    for (case_no, case) in cases.iter().enumerate() {
        let original_hashes: Vec<Vec<usize>> =
            serde_json::from_value(case["original_hash_ids"].clone())?;
        if original_hashes.len() != reference_hashes[case_no].len() {
            return Err(format!("layer {layer} case {case_no} sequence length mismatch").into());
        }
        let mut expected_compact = Vec::with_capacity(original_hashes.len() * HASH_COLS);
        for (pos, rows) in original_hashes.iter().enumerate() {
            if rows.len() != HASH_COLS || reference_hashes[case_no][pos][0].len() != HASH_COLS {
                return Err(format!("layer {layer} case {case_no} hash width mismatch").into());
            }
            let layer_index = if layer == 1 { 0 } else { 1 };
            if rows != &reference_hashes[case_no][pos][layer_index] {
                return Err(format!(
                    "layer {layer} case {case_no} original indices differ from full hash oracle at position {pos}"
                )
                .into());
            }
            for &row in rows {
                expected_compact.push(
                    *row_to_compact
                        .get(&row)
                        .ok_or_else(|| format!("layer {layer} missing compact row for {row}"))?,
                );
            }
        }
        let got_compact = read_i64(&dir.join(format!("case{case_no}_indices.bin")))?;
        if got_compact != expected_compact {
            let first = got_compact
                .iter()
                .zip(&expected_compact)
                .enumerate()
                .find(|(_, (a, b))| a != b);
            return Err(format!(
                "layer {layer} case {case_no} compact indices mismatch: {first:?}"
            )
            .into());
        }

        let seq = original_hashes.len();
        let mut gathered = vec![0.0f32; seq * HASH_COLS * 256];
        for token in 0..seq {
            for col in 0..HASH_COLS {
                engram.embed.row_into(
                    got_compact[token * HASH_COLS + col],
                    &mut gathered
                        [(token * HASH_COLS + col) * 256..(token * HASH_COLS + col + 1) * 256],
                );
            }
        }
        gathered.iter_mut().for_each(|v| *v = bf16_roundtrip(*v));
        compare_bf16(
            &format!("layer={layer} case={case_no} gathered"),
            &gathered,
            &read_bf16(&dir.join(format!("case{case_no}_gathered.bin")))?,
        )?;

        let mut projected = Vec::with_capacity(seq * (DIM * (HC_MULT + 1)));
        for token in 0..seq {
            let input = &gathered[token * HASH_COLS * 256..(token + 1) * HASH_COLS * 256];
            let mut row = vec![0.0f32; DIM * (HC_MULT + 1)];
            engram.wkv.matvec(input, &mut row, None);
            row.iter_mut().for_each(|v| *v = bf16_roundtrip(*v));
            projected.extend_from_slice(&row);
        }
        compare_bf16(
            &format!("layer={layer} case={case_no} projected"),
            &projected,
            &read_bf16(&projected_root.join(format!("layer-{layer}/case{case_no}_projected.bin")))?,
        )?;

        let input = read_bf16(&dir.join(format!("case{case_no}_input.bin")))?;
        let expected_output = read_bf16(&dir.join(format!("case{case_no}_output.bin")))?;
        let mask: Option<Vec<bool>> = if case["token_mask"].is_null() {
            None
        } else {
            Some(serde_json::from_value(case["token_mask"].clone())?)
        };
        let mut output = Vec::with_capacity(input.len());
        for token in 0..seq {
            let mut h = input[token * HC_MULT * DIM..(token + 1) * HC_MULT * DIM].to_vec();
            dsv41_apply_engram_for_test(
                &engram,
                &mut h,
                &got_compact[token * HASH_COLS..(token + 1) * HASH_COLS],
                &cfg(),
                mask.as_ref().map(|m| m[token]).unwrap_or(true),
            );
            output.extend_from_slice(&h);
        }
        compare_bf16(
            &format!("layer={layer} case={case_no} output"),
            &output,
            &expected_output,
        )?;
        println!(
            "component layer={} case={} name={} compact_indices=true gathered=true projected=true output=true",
            layer,
            case_no,
            case["name"].as_str().unwrap_or("?")
        );
    }
    Ok(())
}

pub fn model_dtype(&self) -> Option<TensorDtype>

The layout this tensor is stored in, when it is mapped from a model. The frames branch on it — a q2tp gate against a q4tp down is a real combination in the 2-bit profile and needs a different kernel.

pub fn model_idx(&self) -> Option<usize>

The tensor’s index in the model directory, when it is mapped from one. The GPU frames bind by index rather than by name — a name lookup per layer per token is not free, and the index is what the device cache is keyed on anyway.

pub fn model_arc(&self) -> Option<Arc<CmfModel>>

The model this tensor is mapped from, when it is mapped at all. The GPU frames need the container to reach the bytes; a QTensor already holds it, and threading a second handle down every call site to say the same thing invites the two to disagree.

pub fn rows(&self) -> usize

pub fn mapped_q4tp(&self) -> Option<(&Arc<CmfModel>, usize)>

Same slot as mapped_q4t for a q4tp tensor — the fused DiT FFN picks its kernels by which of the two answers.

pub fn mapped_device_gemm(&self) -> Option<(&Arc<CmfModel>, usize)>

(model, tensor idx) for a mapped weight in ANY codec the fused device paths can run — four-bit tiled or either int8 layout.

The fused DiT chains asked for mapped_q4tp by name, so an eight-bit container never reached them and rendered through per-op GEMMs even after those kernels learned its codec. The gate is what the codec has a device GEMM for, not which codec it is.

pub fn mapped_q2tp(&self) -> Option<(&Arc<CmfModel>, usize)>

(model, tensor idx) for a q2tp mapped weight — the 2-bit twin of mapped_q4tp, used by the mixed MoE profile.

pub fn cols(&self) -> usize

pub fn mapped_q1(&self) -> Option<(&Arc<CmfModel>, usize)>

(model, tensor idx) for a q1 mapped weight — the wgpu token graph keys its resident VRAM cache by idx. None for any other dtype/kind.

pub fn graph_weight(&self) -> Option<(&Arc<CmfModel>, usize, u8, &[f32])>

(model, idx, kind, row_scale) for a graph-capable mapped weight. kind: 0=q8_row (per-row scales), 1=q1, 2=q4_block, 3=q1t (tile-embedded, no rs), 5=q4_tiled, 6=q4tp, 7=q8_2f (both scale planes live inside the tensor). None only for vbit.

The old comment here claimed q4_block was unhandled while the arm right below mapped it, and it named q8_2f as unhandled after that stopped being true — a stale comment on this function is how a model silently loses the graph, so it is worth keeping honest.

pub fn as_f32(&self) -> Option<&[f32]>

Dense f32 view — only for owned tensors. Masked/sparse execution paths require it; quantized weights don’t support masks yet.

pub fn row_f32(&self, r: usize, dst: &mut [f32])

Dequantize one row into dst (embedding lookup).

pub fn sparse_col_ok(&self) -> bool

Can this tensor’s columns be read cheaply (for sparse down_proj)? True for F32/Q8Row/Q8_2f (per-row scale, direct strided access); false for group-packed q4/vbit (column access would unpack whole groups — sparse execution falls back to f32 for those).

pub fn add_col_scaled(&self, c: usize, w: f32, out: &mut [f32])

down_proj [hidden, inter]: accumulate w · col(c) into out [hidden] — reads ONLY column c (one neuron) from the mmap, no full-matrix dequant. out[k] += w · down[k, c].

pub fn prefetch_row(&self, r: usize)

Touch the head of row r so the DRAM latency of the next neuron’s weights overlaps the current one’s arithmetic.

Scattered rows are what per-token sparsity reads, and a 2 KB stride is past what the hardware prefetcher follows: without this every row starts with a cold miss that nothing hides. One touch per 512 bytes is enough — the rest of the row is a sequential run the prefetcher does pick up.

pub fn add_row_scaled( &self, r: usize, w: f32, out: &mut [f32], scratch: &mut [f32], )

out += w · row(r) — the transposed twin of add_col_scaled.

A neuron’s down weights are a COLUMN of [hidden, inter], and a column is strided: reading one costs a cache line per element, so per-neuron dynamic sparsity saves arithmetic and no bytes. Stored transposed (down_proj.t.weight, [inter, hidden]) the same weights are a contiguous ROW, and this accumulate reads exactly the neurons the token asked for.

pub fn row_dot(&self, r: usize, x: &[f32], scratch: &mut [f32]) -> f32

Dot of row r with x (gate/up active-neuron path). Reads only row r from the mmap — no full dequant. q4/vbit dequant the row into scratch first (rare for active-FFN weights).

pub fn matvec(&self, x: &[f32], out: &mut [f32], pool: Option<&Pool>)

out = W · x (row-major). F32 delegates to the historical bit-exact path; Mapped runs the fused int8 kernel.

Examples found in repository

examples/matvec_bw.rs (line 58)

fn main() {
    let mut args = std::env::args().skip(1);
    let path = args
        .next()
        .expect("usage: matvec_bw <model.cmf> [sweep|one <tensor>]");
    let mode = args.next().unwrap_or_else(|| "sweep".to_string());
    let model = Arc::new(cortiq_core::CmfModel::open(&path).expect("open model"));

    // Real LM activations carry a few heavy channels (>8·rms); measured
    // mean on this model is ~3.7. NOUT models that distribution.
    let nout: usize = std::env::var("NOUT")
        .ok()
        .and_then(|v| v.parse().ok())
        .unwrap_or(4);
    let mk_x = |cols: usize| -> Vec<f32> {
        let mut x: Vec<f32> = (0..cols).map(|i| ((i % 17) as f32 - 8.0) / 8.0).collect();
        for k in 0..nout {
            x[k * 37 % cols] = 40.0;
        }
        x
    };

    if mode == "one" {
        let name = args
            .next()
            .unwrap_or_else(|| "model.embed_tokens.weight".to_string());
        let entry = model.tensor(&name).expect("tensor not found");
        let (rows, cols) = (entry.shape[0], entry.shape[1]);
        let nbytes = entry.nbytes as f64;
        println!(
            "tensor {name}: {rows}x{cols} {:?} = {:.1} MB, NOUT={nout}",
            entry.dtype,
            nbytes / 1e6
        );
        let t = QTensor::from_model(&model, &name).expect("wrap");
        let x = mk_x(cols);
        let mut out = vec![0f32; rows];
        t.matvec(&x, &mut out, None);
        for nt in [1usize, 2, 4, 6, 8, 10] {
            let pool = if nt == 1 { None } else { Some(Pool::new(nt)) };
            let iters = 8;
            t.matvec(&x, &mut out, pool.as_ref());
            let t0 = Instant::now();
            for _ in 0..iters {
                t.matvec(&x, &mut out, pool.as_ref());
            }
            let el = t0.elapsed().as_secs_f64();
            println!(
                "threads={nt:2}  {:6.2} ms/matvec  {:6.1} GB/s (sink {:.3})",
                el / iters as f64 * 1e3,
                nbytes * iters as f64 / el / 1e9,
                out[0]
            );
        }
        return;
    }

    // sweep: every 2-D q8_2f tensor once = one decode's worth of weights.
    let names: Vec<String> = model
        .tensors
        .iter()
        .filter(|t| t.dtype == TensorDtype::Q8_2f && t.shape.len() == 2)
        .map(|t| t.name.clone())
        .collect();
    let total_bytes: f64 = model
        .tensors
        .iter()
        .filter(|t| t.dtype == TensorDtype::Q8_2f && t.shape.len() == 2)
        .map(|t| t.nbytes as f64)
        .sum();
    println!(
        "sweep: {} q8_2f tensors, {:.2} GB total (= weights streamed per decode token), NOUT={nout}",
        names.len(),
        total_bytes / 1e9
    );

    let tensors: Vec<(QTensor, Vec<f32>, Vec<f32>)> = names
        .iter()
        .map(|n| {
            let e = model.tensor(n).unwrap();
            let (rows, cols) = (e.shape[0], e.shape[1]);
            (
                QTensor::from_model(&model, n).expect("wrap"),
                mk_x(cols),
                vec![0f32; rows],
            )
        })
        .collect();
    let mut tensors = tensors;

    // Whole-model residency pass BEFORE any timing: the first touch of a
    // 4.2 GB mmap faults ~260k pages, which would otherwise be charged to
    // whichever thread count happens to run first. REVERSE=1 flips the
    // order as a check that no first-touch cost is left in the table.
    for _ in 0..2 {
        for (t, x, out) in tensors.iter_mut() {
            t.matvec(x, out, None);
        }
    }
    let mut counts = vec![1usize, 2, 4, 6, 8, 10];
    if std::env::var("REVERSE").is_ok() {
        counts.reverse();
    }
    for nt in counts {
        let pool = if nt == 1 { None } else { Some(Pool::new(nt)) };
        // one warm pass, then two measured
        for (t, x, out) in tensors.iter_mut() {
            t.matvec(x, out, pool.as_ref());
        }
        let iters = 2;
        let t0 = Instant::now();
        for _ in 0..iters {
            for (t, x, out) in tensors.iter_mut() {
                t.matvec(x, out, pool.as_ref());
            }
        }
        let el = t0.elapsed().as_secs_f64();
        let per_tok = el / iters as f64;
        println!(
            "threads={nt:2}  {:7.1} ms/sweep  {:6.1} GB/s  -> weight-path-only ceiling {:5.1} tok/s",
            per_tok * 1e3,
            total_bytes * iters as f64 / el / 1e9,
            1.0 / per_tok
        );
    }
}

examples/dsv41_engram_proof.rs (line 426)

fn verify_component_layer(
    root: &Path,
    projected_root: &Path,
    layer: usize,
    reference_hashes: &[Vec<Vec<Vec<usize>>>],
) -> Result<(), Box<dyn Error>> {
    let dir = root.join(format!("raw-layer-{layer}"));
    let layer_meta: Value =
        serde_json::from_slice(&fs::read(root.join(format!("layer-{layer}.json")))?)?;
    let original_rows: Vec<usize> = serde_json::from_value(layer_meta["original_row_ids"].clone())?;
    let row_to_compact: HashMap<usize, usize> = original_rows
        .iter()
        .copied()
        .enumerate()
        .map(|(compact, original)| (original, compact))
        .collect();
    let prefix = format!("model.layers.{layer}.engram");
    let cmf = dir.join("engram-component.cmf");
    let model = Arc::new(CmfModel::open(&cmf)?);
    let embed = RawFp8Rows::from_model(
        &model,
        &format!("{prefix}.embed.weight"),
        &format!("{prefix}.embed.scale"),
    )?;
    let wkv = QTensor::from_model(&model, &format!("{prefix}.wkv.weight"))?;
    let q_weight = f32_tensor(&model, &format!("{prefix}.q_weight"))?;
    let k_weight = f32_tensor(&model, &format!("{prefix}.k_weight"))?;
    let engram = Dsv41Engram {
        embed,
        wkv,
        q_weight,
        k_weight,
    };
    let cases = layer_meta["cases"]
        .as_array()
        .ok_or("layer cases missing")?;
    for (case_no, case) in cases.iter().enumerate() {
        let original_hashes: Vec<Vec<usize>> =
            serde_json::from_value(case["original_hash_ids"].clone())?;
        if original_hashes.len() != reference_hashes[case_no].len() {
            return Err(format!("layer {layer} case {case_no} sequence length mismatch").into());
        }
        let mut expected_compact = Vec::with_capacity(original_hashes.len() * HASH_COLS);
        for (pos, rows) in original_hashes.iter().enumerate() {
            if rows.len() != HASH_COLS || reference_hashes[case_no][pos][0].len() != HASH_COLS {
                return Err(format!("layer {layer} case {case_no} hash width mismatch").into());
            }
            let layer_index = if layer == 1 { 0 } else { 1 };
            if rows != &reference_hashes[case_no][pos][layer_index] {
                return Err(format!(
                    "layer {layer} case {case_no} original indices differ from full hash oracle at position {pos}"
                )
                .into());
            }
            for &row in rows {
                expected_compact.push(
                    *row_to_compact
                        .get(&row)
                        .ok_or_else(|| format!("layer {layer} missing compact row for {row}"))?,
                );
            }
        }
        let got_compact = read_i64(&dir.join(format!("case{case_no}_indices.bin")))?;
        if got_compact != expected_compact {
            let first = got_compact
                .iter()
                .zip(&expected_compact)
                .enumerate()
                .find(|(_, (a, b))| a != b);
            return Err(format!(
                "layer {layer} case {case_no} compact indices mismatch: {first:?}"
            )
            .into());
        }

        let seq = original_hashes.len();
        let mut gathered = vec![0.0f32; seq * HASH_COLS * 256];
        for token in 0..seq {
            for col in 0..HASH_COLS {
                engram.embed.row_into(
                    got_compact[token * HASH_COLS + col],
                    &mut gathered
                        [(token * HASH_COLS + col) * 256..(token * HASH_COLS + col + 1) * 256],
                );
            }
        }
        gathered.iter_mut().for_each(|v| *v = bf16_roundtrip(*v));
        compare_bf16(
            &format!("layer={layer} case={case_no} gathered"),
            &gathered,
            &read_bf16(&dir.join(format!("case{case_no}_gathered.bin")))?,
        )?;

        let mut projected = Vec::with_capacity(seq * (DIM * (HC_MULT + 1)));
        for token in 0..seq {
            let input = &gathered[token * HASH_COLS * 256..(token + 1) * HASH_COLS * 256];
            let mut row = vec![0.0f32; DIM * (HC_MULT + 1)];
            engram.wkv.matvec(input, &mut row, None);
            row.iter_mut().for_each(|v| *v = bf16_roundtrip(*v));
            projected.extend_from_slice(&row);
        }
        compare_bf16(
            &format!("layer={layer} case={case_no} projected"),
            &projected,
            &read_bf16(&projected_root.join(format!("layer-{layer}/case{case_no}_projected.bin")))?,
        )?;

        let input = read_bf16(&dir.join(format!("case{case_no}_input.bin")))?;
        let expected_output = read_bf16(&dir.join(format!("case{case_no}_output.bin")))?;
        let mask: Option<Vec<bool>> = if case["token_mask"].is_null() {
            None
        } else {
            Some(serde_json::from_value(case["token_mask"].clone())?)
        };
        let mut output = Vec::with_capacity(input.len());
        for token in 0..seq {
            let mut h = input[token * HC_MULT * DIM..(token + 1) * HC_MULT * DIM].to_vec();
            dsv41_apply_engram_for_test(
                &engram,
                &mut h,
                &got_compact[token * HASH_COLS..(token + 1) * HASH_COLS],
                &cfg(),
                mask.as_ref().map(|m| m[token]).unwrap_or(true),
            );
            output.extend_from_slice(&h);
        }
        compare_bf16(
            &format!("layer={layer} case={case_no} output"),
            &output,
            &expected_output,
        )?;
        println!(
            "component layer={} case={} name={} compact_indices=true gathered=true projected=true output=true",
            layer,
            case_no,
            case["name"].as_str().unwrap_or("?")
        );
    }
    Ok(())
}

pub fn matvec2( &self, x1: &[f32], x2: &[f32], o1: &mut [f32], o2: &mut [f32], pool: Option<&Pool>, )

Fused two-input matvec (MTP verify pair): weights streamed once.

impl QTensor

pub fn q4tp_mapped(&self) -> Option<(&Arc<CmfModel>, usize)>

Batched matvec (prefill-GEMM): xs — row-major [b, cols], out — row-major [b, rows]. Element-wise semantics are IDENTICAL to b matvec calls (same dot kernels in the same order); the win — the weight row streams from DRAM once per batch, not b times. (model, index) when this is a memory-mapped q4tp tensor — the identity a device-resident chain needs to hand tp_matmat the weight without going through this struct’s own dispatch.

pub fn matmat( &self, xs_all: &[f32], b: usize, out: &mut [f32], pool: Option<&Pool>, )

impl QTensor

pub fn device_matmat(&self, xs: &[f32], b: usize, out: &mut [f32]) -> bool

The device GEMM this tensor would take, run once on the caller’s data — the startup parity probe’s arm, and the one place that knows which entry point each codec has.

It exists because the probe used to look for a q4tp weight by name AND dtype, and a container packed any other way was declared “host path” for the whole render even though its codec had a device GEMM of its own. A gate that only recognizes one codec is a gate that silently downgrades every other one.

pub fn matvec_many<const N: usize>( ts: [&QTensor; N], x: &[f32], outs: [&mut [f32]; N], pool: Option<&Pool>, )

Multi-matrix job (roadmap §3 P0): N tensors sharing one input run under a SINGLE pool dispatch — QKV or gate+up cost one barrier instead of N. Per-row math is the exact same kernel as matvec (bit-identical outputs); only the dispatch is fused. Falls back to N sequential matvecs when the set is not a uniform q8-family/F32 group or there is no pool.

impl QTensor

pub fn matvec2_many<const N: usize>( ts: [&QTensor; N], x1: &[f32], x2: &[f32], o1s: [&mut [f32]; N], o2s: [&mut [f32]; N], pool: Option<&Pool>, )

Pair-input multi-matrix job: N tensors × 2 shared inputs under a single pool dispatch — the MTP/pair decode path publishes one job for Q/K/V (and one for gate+up) instead of one per tensor. Per-row math is exactly matvec2’s kernels; bit-identical.

pub fn matvec_silu_mul( gate: &QTensor, up: &QTensor, x: &[f32], out: &mut [f32], pool: Option<&Pool>, ) -> bool

Fused gate+up matvec with SiLU·mul: for each row r, computes silu(gate·x) * (up·x) and writes to out[r]. ONE pool dispatch, no intermediate g/u buffers, no separate silu pass. Falls back (returns false) for unsupported dtype combos.

pub fn matvec_silu_mul_limited( gate: &QTensor, up: &QTensor, x: &[f32], out: &mut [f32], limit: f32, pool: Option<&Pool>, ) -> bool

Fused gate+up+SiLU with the GLM asymmetrical clamp. limit == 0 preserves the historical unclamped helper; a positive limit clamps up to both sides and gate only from above, matching the GLM SwiGLU reference. Keeping the limit in the row kernel avoids the two intermediate vectors and the extra combine pass on the Q2TP experts.

pub fn moe_gate_up_many( pairs: &[(&QTensor, &QTensor)], x: &[f32], outs: &mut [Vec<f32>], pool: Option<&Pool>, ) -> bool

Every routed expert’s fused gate/up/SiLU under ONE pool dispatch.

The per-expert path pays a pool barrier per expert per stage: at 9 experts over 40 layers that is ~720 barriers a token, and a decode profile of Qwen3.6-35B-A3B showed the pool parked in psynch_cvwait about twice as long as it spent computing. Laying every expert’s rows end-to-end in one virtual row space collapses the stage to a single dispatch. The per-row body is the single-expert q4tp arm verbatim, so outputs are bit-identical.

false = something is outside the fused q4tp kernel (dtype, shape, or the CMF_SDOT=0 exact contract); the caller walks the ordinary per-expert path.

pub fn moe_down_many( downs: &[&QTensor], gs: &[Vec<f32>], weights: &[f32], out: &mut [f32], pool: Option<&Pool>, ) -> bool

Every routed expert’s down projection, weighted and summed into out, under ONE pool dispatch.

Partitioned by OUTPUT row rather than by expert: each row is owned by a single worker, so the experts are summed in the caller’s order — the same sequence of f32 adds the serial out[i] += w·eo[i] loop performs, hence bit-identical. Partitioning by expert instead would race on the shared accumulator.