Expand description
Persistent worker pool for row-parallel matvecs.
Threads are spawned once and spin-then-park between calls — vmfcore measured spawn-per-matvec at ~+27% decode cost versus a persistent pool. Parallelism is by disjoint row ranges, so results are bit-identical to the serial path (each row’s dot product is computed the same way).
Dispatch is a single shared job slot + atomic epoch (roadmap §3 P0):
the caller publishes one pointer, bumps the epoch and JOINS THE WORK
as the extra worker instead of blocking on a latch. The previous
design allocated an Arc<Latch> and pushed a message into every
worker’s mpsc channel for every matvec (~200 dispatches/token) —
with decode-grade matvecs that synchronization was its own budget.
Workers spin for CMF_POOL_SPIN iterations before parking.
Default 4000: at ~39 dispatches/token, park-immediately pays the
unpark syscall on every worker for every dispatch — measured on an
M4 (interleaved A/B, current epoch dispatch + parked-flag design):
Qwen-0.5B q8 decode 101→115 tok/s, q4t 117→149, the 50M bench model
549→954 at spin=4000 vs spin=0. An early measurement that showed
spinning LOSING (−25% on q8) predates the parked-flag skip and the
multi-matrix dispatch cuts; it no longer reproduces. Over-spinning
still hurts (200k: −15% vs 4k — spinners steal the caller’s serial
cycles), so the budget stays bounded. CMF_POOL_SPIN=0 restores
park-immediately for share-the-box serving.
CMF_THREADS env: 0/1 = serial, N = worker count
(default: available_parallelism − 1, capped at 8).
Structs§
- Pool
- Persistent thread pool: shared job slot, epoch dispatch, caller participation.
Functions§
- dispatch_
count - Total pool jobs published since process start (all pools).
- matvec_
rows - Row-parallel dense matvec:
out[o] = Σ_j w[o·in + j]·x[j]. Bit-identical to the serial loop (row order does not change math). - matvec_
rows2 - Two-input row matvec: one pass over the weight rows serves BOTH inputs — CPU decode is memory-bound, so the second position costs a fraction of the first (this is where MTP speculative verify wins). Per-output accumulation order matches the single-input path exactly → bit-identical results.