Module threads

Expand description

CPU worker-pool policy, shared by ferrox (CLI) and ferrox-server.

Two things this module exists to control, both of which were measured to matter far more than any kernel change on Apple Silicon:

Thread count. available_parallelism() on an M2 Pro reports 10 (6 performance + 4 efficiency cores). Splitting a decode GEMV across all 10 makes every fork-join wait on the slowest E-core slice. llama.cpp defaults to hw.perflevel0.physicalcpu for exactly this reason (common_cpu_get_num_math), and collapses when forced above it – 346 -> 176 tok/s on SmolLM2-135M Q8_0 going from -t 4 to -t 10. So default to the performance-core count, not the logical-core count.
Thread QoS. macOS schedules threads onto E-cores based on their Quality-of-Service class, and QoS is inherited from whichever thread spawned them. Rayon builds its global pool lazily, on first use – which inside ferrox-server is a Tokio spawn_blocking task, not the main thread. If that blocking thread carries a demoted QoS, every rayon worker inherits it and the whole matvec runs on efficiency cores. init_cpu_pool pins the workers to USER_INTERACTIVE explicitly so the pool’s placement does not depend on who happened to touch rayon first.
Dedicated GEMV pool. Row-parallel matvec runs on a crate-owned rayon::ThreadPool, not rayon’s global pool, so library consumers and Tokio blocking threads do not fight over thread count or scheduling. See for_each_row.

Functions§

current_qos_name: The calling thread’s macOS QoS class, as a human-readable name. None off macOS, where the concept does not exist.
for_each_chunk_init: Chunk-parallel sibling of for_each_row; chunks are never split.
for_each_row: One output row per slot; parallel when should_parallelize and the GEMV pool has more than one worker. Rows are never split.
gemv_num_threads: Active GEMV pool width, or 1 when the pool is unavailable.
init_cpu_pool: Builds the global rayon pool with an explicit width and an explicit QoS, so neither depends on which thread first touched rayon. Safe to call more than once and from either binary; a pool that already exists is left alone.
init_gemv_pool: Eagerly build the dedicated GEMV pool (no-op if already built).
perf_core_count: Number of performance cores, which is the useful width for a decode GEMV. On macOS this is hw.perflevel0.physicalcpu (llama.cpp reads the same key). Elsewhere, and if the query fails, falls back to available_parallelism.
resolve_cpu_threads: How many rayon workers to run: FERROX_CPU_THREADS, else RAYON_NUM_THREADS, else perf_core_count.
resolve_gemv_threads: GEMV pool width: FERROX_GEMV_THREADS, else resolve_cpu_threads.
should_parallelize: Prefer serial when fork-join overhead exceeds the matvec work. ~256k element-ops matches the previous prefer_serial_matvec gate.