Skip to main content

with_row_exact

Function with_row_exact 

Source
pub fn with_row_exact<R>(f: impl FnOnce() -> R) -> R
Expand description

Runs f with row-exact products on this thread: quantized and gathered products of up to ROW_INDEPENDENT_ROWS rows (tokens, for gathers) evaluated or built inside give each row the bits of its one-row product, through one matrix-vector product per row (or nuvec’s few-row kernels with the same arithmetic). Outside it they route as MLX does, whose multi-vector kernels are faster from 2 rows on but give a row other bits than its one-row product. Speculative verification runs in this mode, and so can batched serving that must equal each request run alone.