Struct Pool

Source

pub struct Pool { /* private fields */ }

Expand description

Persistent thread pool: shared job slot, epoch dispatch, caller participation.

Implementations§

Source §

impl Pool

Source

pub fn new(n_workers: usize) -> Self

Source

pub fn with_spin(n_workers: usize, spin_budget: usize) -> Self

Explicit spin budget (tests pin it without touching the env).

Source

pub fn effective_threads() -> usize

The thread count from_env would use RIGHT NOW: forced (C ABI)

CMF_THREADS > big-core topology > available_parallelism−1. ≤1 means the model runs serial (no pool). Introspection (execution_mode, status endpoints) must report THIS, not available_parallelism.

Source

pub fn from_env() -> Option<Arc<Self>>

Pool sized from CMF_THREADS (see module docs). None = serial. Without the env, heterogeneous ARM defaults to its BIG cores.

Source

pub fn n_workers(&self) -> usize

Spawned worker threads (the caller joins each job on top).

Source

pub fn bind_numa(&self, regions: &[&[u8]])

Keep the pool on the NUMA node that holds regions (the model’s weight bytes). Linux with two or more nodes only; CMF_NUMA=0 turns it off, CMF_NUMA=node:<n> forces a node.

WHY: decode streams every weight once per token, and on a two-socket host the page cache holds a file on whichever node read it. Unpinned, the scheduler spreads the workers over both sockets and half the matvec rows cross the socket link. Measured on a 2×EPYC 7763 pod with the model’s pages all on node 0 (31 CPUs of cgroup quota): a STREAM-style read over a node-0 buffer gives 42 GB/s from 31 unpinned threads and 74 GB/s from 31 threads kept on node 0. The mask is the node’s physical cores (first SMT sibling) when there are enough of them for the pool, else the whole node; never narrower than the pool, so nothing oversubscribes. Threads are bound to a SET of cores, not to one core each: the scheduler still balances inside the node. The calling thread adopts the same mask on its next dispatch.

Source

pub fn two_nodes(&self) -> bool

The odd workers run on a second NUMA node (bind_two_nodes).

Source

pub fn far_worker(&self, w: usize) -> bool

Worker w of a job runs on the far node of a two-node pool (the caller, participant n_workers(), never does).

Source

pub fn bind_two_nodes( &self, hint: Option<usize>, ) -> Option<(usize, usize, usize, usize)>

Put the odd workers on a second NUMA node, the even ones and the caller staying where they are: (home node, far node, threads at home with the caller, threads on the far node). Linux with two or more nodes; the home node is the one bind_numa chose, or hint where it chose none (a file spread over the nodes), and the node’s mask is then published for every pool’s callers as bind_numa does.

WHY: a job that streams weights is bound by the node’s memory bus, not by its cores. Measured on 2 x EPYC 9554 under a 31-CPU quota: 28 threads on one node read 153-180 GB/s (the node gives 220-230), 14 + 14 over both nodes with each half’s data on its own node read 393-421 GB/s. It only pays when the job gives the far workers data that lives on their node, so the caller places it there (the far workers read it first) and asks far_worker for the side of a participant; unbind_two_nodes takes it back.

Source

pub fn unbind_two_nodes(&self)

Take the odd workers back to the home node (bind_two_nodes).

Source

pub fn bind_node(&self, node: usize)

Keep the pool’s workers on the node of the published mask, or on node where nothing published one (bind_numa chose no node: a file spread over them); the caller adopts the same mask.

Source

pub fn place_file_pages( &self, home: &[&[u8]], far: &[&[u8]], nodes: (usize, usize), fill: usize, on_move: &dyn Fn(usize), ) -> Option<NodePlacement>

Put the page-cache pages behind home on the home node of a two-node pool and the ones behind far on its far node (nodes, as bind_two_nodes returned them); both are spans of mapped files. None where that cannot be done or told: no two-node pool, not Linux, spans that are no whole pages of a file this process can open.

WHY: a file’s pages stay on the node whose thread read them first, and a container gets no call that moves them (move_pages and mbind are refused under RunPod’s seccomp profile). What it does get: drop a span from the page cache (posix_fadvise(DONTNEED)) and read it again from a thread of the wanted node. Measured on 2 x EPYC 9554: a 1 GB range that sat 85-88 % on node 0 came back 98.6-99.8 % on node 1, read by 14 threads at 22 GB/s.

A side is read again only when under 90 % of a sample of it sits on its node (on_move hears the bytes first: on a slow disk it takes a while), so a second start moves nothing. Of the first fill spans of each side, the ones the cache does not hold at all are read by their node before anything else places them.

Source

pub fn run_rows(&self, rows: usize, f: &(dyn Fn(usize, usize) + Sync))

Run f(row_start, row_end) over 0..rows, self-balancing.

One dispatch, but workers pull row-ranges from a shared cursor instead of each taking a fixed 1/n slice. On a heterogeneous CPU (Apple Silicon: 4 P-cores + 6 E-cores here) a static split makes every matvec end at the SLOWEST core’s pace while the fast ones idle at the barrier; pulling by grain lets a P-core take several chunks for each one an E-core takes, so skew collapses to a single grain. Row ranges stay disjoint and each row’s dot is computed exactly as in the serial path → bit-identical output.

Source

pub fn run_many(&self, parts: &[(usize, &(dyn Fn(usize, usize) + Sync))])

Multi-matrix job: one dispatch serves SEVERAL row spaces (roadmap §3 P0 — «одна внешняя публикация job на слой»). Parts are laid out back-to-back in a virtual row space and pulled by grain from one shared cursor, so QKV or gate+up cost a single barrier instead of one each. Each part’s f(start, end) sees its OWN row indices — per-row math and outputs are bit-identical to separate run_rows calls.

Source

pub fn run(&self, f: &(dyn Fn(usize, usize) + Sync))

Run f(worker_idx, n_participants) on every worker AND the calling thread (worker_idx = n_workers() for the caller); returns when all participants have finished.