Skip to main content

q4tp_matvec

Function q4tp_matvec 

Source
pub fn q4tp_matvec(
    model: &Arc<CmfModel>,
    idx: usize,
    xs: &[f32],
    rows: usize,
    cols: usize,
    out: &mut [f32],
) -> bool
Expand description

Single-token q4tp matvec on the device — the lm_head class. Through the DEDICATED matvec kernel: the batched GEMM at b=1 measured 11.73 ms against the host’s 9.51 on the release head, so the route that was supposed to save eleven milliseconds a token lost its own probe instead.