Expand description
Persistent worker pool for row-parallel matvecs.
Threads are spawned once and spin-then-park between calls — vmfcore measured spawn-per-matvec at ~+27% decode cost versus a persistent pool. Parallelism is by disjoint row ranges, so results are bit-identical to the serial path (each row’s dot product is computed the same way).
Dispatch is a single shared job slot + atomic epoch (roadmap §3 P0):
the caller publishes one pointer, bumps the epoch and JOINS THE WORK
as the extra worker instead of blocking on a latch. The previous
design allocated an Arc<Latch> and pushed a message into every
worker’s mpsc channel for every matvec (~200 dispatches/token) —
with decode-grade matvecs that synchronization was its own budget.
Workers spin for CMF_POOL_SPIN iterations before parking.
Default 0 (park immediately): on Apple Silicon unpark is cheap and
measured A/B showed spinning workers stealing cycles from the
caller’s serial sections (q8 decode 299–345 tok/s at spin=6000 vs
406–412 at spin=0 on the 50M bench model). Set CMF_POOL_SPIN>0 to
experiment on platforms with slower futex wakeups.
CMF_THREADS env: 0/1 = serial, N = worker count
(default: available_parallelism − 1, capped at 8).
Structs§
- Pool
- Persistent thread pool: shared job slot, epoch dispatch, caller participation.
Functions§
- dispatch_
count - Total pool jobs published since process start (all pools).
- matvec_
rows - Row-parallel dense matvec:
out[o] = Σ_j w[o·in + j]·x[j]. Bit-identical to the serial loop (row order does not change math). - matvec_
rows2 - Two-input row matvec: one pass over the weight rows serves BOTH inputs — CPU decode is memory-bound, so the second position costs a fraction of the first (this is where MTP speculative verify wins). Per-output accumulation order matches the single-input path exactly → bit-identical results.