threadteam
Splittable CPU placement token for the tensor4all CPU stack. threadteam is the one place
that starts tasks: kernels take an Exec token and split it, and no scheduler
type appears in any public signature.
Used by strided-rs, tcontract-rs, cpueinsum-rs, tlinalg-rs and tenferro-rs.
Design records: docs/design/placement.md (the token),
docs/design/waiting.md (how the workers wait).
The workers belong to a private Rayon pool per Pool. A pool with the default
thread settings keeps its Rayon pool in a process-wide cache when it is
dropped, and the next pool of the same width takes it, so a pool built per call
(faer's with_team) reuses running threads instead of starting new ones.
Usage
use ;
use ;
// The owner builds a pool once and keeps it.
let pool = builder.threads.build?;
// Per call, the owner mints a token for `budget` slots of the pool.
let mut exec = new?;
// Several phases in sequence: reborrow, because entries consume the token.
let total = new;
exec.reborrow.for_ranges;
// Independent parts: each job gets a child token with its share of the slots.
exec.fork;
# Ok::
Exec::serial() needs no pool and runs everything inline.
Intended consumer pattern
The owner of the CPU resources (an engine, a contraction resource bundle) holds
a threadteam::Pool and mints Exec::new(&pool, budget) for each call, passing
the token to kernels by value. Kernels never store a token and never mint one.
API
| Item | Role |
|---|---|
Pool, PoolBuilder |
fixed set of workers; threads, thread_name, stack_size (default 16 MiB, sized for recursive LU), on_start, build |
Exec::new(&pool, budget) |
mint a token; the only clamp (to the pool width); budget == 0 is an error |
Exec::serial() |
the calling thread only; const |
Exec::budget, is_serial |
inspect the token (is_serial means budget 1) |
Exec::reborrow(&mut self) |
same pool and budget while the parent is exclusively borrowed |
Exec::limit(max) |
narrow to min(budget, max) |
Exec::fork(jobs, f) |
min(jobs, budget) tasks; shares differ by at most one and sum to the budget; job i gets a child token |
Exec::fork_sized(&shares, f) |
one child per share, child i gets a token of exactly shares[i] slots; shares sum to at most the budget (Err(Error::SharesExceedBudget) otherwise, never clamped; a zero share is Err(Error::ZeroBudget)); leftover slots stay idle and return to the parent after the join; fork is its even special case |
Exec::fork_sized_slots(&shares, &mut slots, f) |
fork_sized where child i also gets exclusive &mut slots[i] (no lock, so a warm call allocates nothing); each child derives its own &mut slots[i] from its index right before its callback, exclusive because each index is dispatched exactly once, and the borrow of slots ends at the join; a slot count differing from the share count is Err(Error::SlotsMismatch); after a child panic is re-raised every slot holds a valid S the caller can use (the panicking child's slot as it left it) |
Exec::for_ranges(len, grain, f) |
at most budget tasks, ranges of at least grain except possibly the last |
Exec::broadcast(f) |
f(i) for i in 0..budget, all concurrent, or Err(Refused) |
Exec::can_broadcast() |
non-consuming: true exactly when broadcast would not refuse right now; ask before building an expensive setup |
Exec::carries_team() |
true when the token has team authority (minted by a Team or a child of one); team() then returns the carried team without acquisition, so a consumer that wants a TeamExec asks this and keeps external for plain tokens |
Exec::team(f) / team_with(over, f) |
reserve budget - 1 awake pool members for a scope and run barrier-bearing sections through TeamExec (run, run_with, split, fork, reborrow), or Err(Refused::Unavailable); see the design record |
Pool::into_team(width) -> Team (owns the pool; IntoTeamError gives it back), Team::exec(budget) / try_exec, Team::pool, Team::into_pool |
persistent reservation for an engine that owns its pool (Pool is not Clone); minted tokens are ordinary Execs whose fork, for_ranges, broadcast, team dispatch on the reserved members; one leader at a time (exec blocks, try_exec -> Refused::Busy); idle members park while reserved |
PoolBuilder::qos(Qos), raise_current_thread_qos(Qos) |
worker QoS on macOS (default UserInitiated); raise the leading thread without ever lowering it |
PoolBuilder::idle_spin, idle_runs_rayon, acquire_timeout; Pool::release_idle |
waiting policy of idle team members (an idle member returns to Rayon after idle_spin or the spin budget, whichever comes first) |
PoolBuilder::spin_polls, Pool::spin_polls |
spin budget of every team wait, in polls (the shape of libgomp's GOMP_SPINCOUNT): a reserved member waiting for work, the leader in a join and a participant in a barrier spin that many polls, yield a few times, then sleep until woken; throttled to 100 polls when the root scope is oversubscribed (more participants than CPUs in the caller's mask). No wait reads a clock or detects descheduling (docs/design/waiting.md) |
PoolBuilder::pin_participants(bool), Pool::pin_participants(), Pool::pin_skips() |
placement of the participants of a root team scope inside the calling thread's affinity set S (Linux; on by default, false turns it off per pool): the leader is pinned to S[0] in exactly two cases, (1) for the life of a root team scope (Exec::team, or a Team token's team: one sched_setaffinity at entry, one at exit, also on unwind) and (2) for one section when a token of a Team dispatches fork / for_ranges / broadcast without opening a scope (pinned for that section, its own mask back when it ends), and keeps its own mask S everywhere else; member k (participant k >= 1) pins itself onto its own CPU of S other than S[0] (by default round-robin over the NUMA nodes S spans, PinStrategy::Spread; PoolBuilder::pin_strategy(PinStrategy::Pack) gives the 0.4-0.5 placement S[k]), and a member restores its own mask before it returns to Rayon or serves foreign work. Nothing wraps: when S has fewer CPUs than participants nobody is placed and Pool::pin_skips() counts the scope. The members' placement belongs to the team and the leader's pin to one scope, or to one section: the tokens of the thread that called Pool::into_team run fork_sized, for_ranges, ... with every participant on its own CPU without a team scope, and that thread keeps its own mask between sections, so into_team returns without changing it and a thread or a vendor pool started there gets all of S; a token used on another thread places that thread only, for the sections it dispatches, and only inside the mask that thread has. A vendor library creates its thread team from the calling thread and a one-CPU caller would confine the whole provider team, so the leader calls it through TeamExec::external, which sets its mask to S for the call and back to {S[0]} after (two sched_setaffinity calls per call, about 2.3 µs on broccoli, paid every call); a member uses Exec::external (which restores the member's mask first). The mask only says where a provider's threads may run, not how many it starts: that is OMP_NUM_THREADS / OPENBLAS_NUM_THREADS / MKL_NUM_THREADS (see tenferro-rs PERFORMANCE_TIPS.md). The NUMA effect of Spread is not verified on this project's hosts (all one NUMA node). For tensor4all/faer#107 and the evo2 cross-L3 stall; give each team its own CPU set (taskset -c), there is no CPU allocator |
DispatchCosts, WidthCosts, PoolBuilder::dispatch_costs, Exec::dispatch_cost(width), TeamExec::dispatch_cost(width) |
cost of one parallel section per kind of token (team, in pool, outside), steady state; one row per HostClass (HostClass::detect(), keyed by kernel dispatch: x86_64-avx512, x86_64-avx2, x86_64-scalar, aarch64-apple, aarch64, generic), DispatchCosts::for_class(class); measured rows for x86_64-avx512 (team anchors at widths 2, 4, 6 and 8) and aarch64-apple, a measured Zen2 team row for x86_64-avx2 (anchors at ten widths from 2 to 64; its plain kinds stay generic), every other class takes the conservative generic row (3 / 15 / 40 us at the anchors 2, 4, 6); a row holds measured anchors (WidthCosts::new, WidthCosts::flat, or WidthCosts::from_anchors for measured anchors up to 64 and beyond), interpolates between them and grows linearly above the largest one, so a width above the anchors is never charged the cost of the largest anchor; DispatchCosts::host_default() is the detected class's row; Duration::MAX when serial; override per pool |
pays_at(est, sections, cost), Exec::pays(est, sections, width), Exec::width_for(est, sections), width_for_by, PAYS |
the one rule: parallel iff est >= PAYS * sections * D(width); width_for minimises sections * D(k) + est / k |
SplitCost, TransferRate, Exec::width_for_split(split), Team::width_for_split(split), width_for_split_by |
the width model with the transfer of a section whose shares read data the caller has just written: minimises sections * D(k) + est / k + moved(k) / rate with moved(k) = resident_bytes * (k - 1) / k, one drain at one aggregate rate per HostClass (8 ns/KiB measured on aarch64-apple, 41 ns/KiB on x86_64-avx512 and on every unmeasured class); resident_bytes: 0 is exactly width_for, and the transfer is never part of the pay decision (docs/design/thresholds.md §3.8) |
Exec::external(f) |
enter the pool once and hand the budget as a plain width to an outside library (faer, a vendor BLAS); how it divides the width is that library's concern |
Contracts
- No duplication.
Execis neitherClonenorCopy(compile-fail doctests), and every parallel entry consumes it. - Slots are conserved. Under one root token the number of tasks running at once is at most its budget at every nesting depth. A parent waiting for its children keeps no slot.
- Lazy and inline. A serial token, budget one, or one task runs inline on the calling thread and makes no scheduler call, so the process-global Rayon pool is never started implicitly.
- Placement by caller. A worker of the target pool spawns into the current
scope without installing; a plain thread or a worker of another pool enters the
target pool once and waits. The crate keeps no ambient thread-local state: the
one value it stores per thread is the placement record (
pin::PLACED, a non-allocatingCell), and nothing else is inherited by work that runs on the calling thread. broadcastrefuses withRefused::ForkChildfor a token derived fromfork(throughreborroworlimittoo), and withRefused::CallerInPoolwhen the caller is a worker of the same pool. Roots, and theirreborrows andlimits, are eligible. Callers must handleRefusedby falling back to a barrier-free path. Concurrent broadcasts on one pool queue in the same order on every worker (no interleaving); participants are the pool's workers0..budget, so blocking work of another token on those workers delays the broadcast.can_broadcastapplies the same rule through one private helper, so the two cannot disagree.- Shutdown. Dropping a
Poolends its team members and waits until they left. A pool with athread_name, anon_starthook, astack_sizeor aqosof its own has dedicated workers, and its drop joins them: whendropreturns, all of them have exited and their thread-local destructors (including state created in anon_starthook) have run. A pool with the default thread settings keeps its idle workers in a process-wide cache keyed by width. The next default pool of that width takes them and gives each worker the building thread's CPU set. A cached set belongs to one pool at a time, and the cache holds at most as many threads as the larger of the available parallelism and the widest pool built so far; it joins the oldest entries beyond that. A pool dropped from one of its own workers cannot wait for itself, so that drop only signals shutdown and returns without joining or caching. Withoutparallelthere is nothing to join. - No scheduler type is public. There is no constructor from, and no accessor
returning, a scheduler pool.
scripts/check-no-public-rayon.shrunscargo public-api -p threadteam(when installed) and fails ifrayon::appears in the output.
Features
| Feature | Effect |
|---|---|
parallel (default) |
run on a private pool. Without it the same types exist, PoolBuilder::build returns a width-1 pool and every entry runs inline |
ledger |
test support: ledger::max_live(root), live, installs, and Exec::ledger_root. Records live tasks per root token and pool installs; compiled out when off |
Consumers assert ledger::max_live(root) <= budget in their own tests, and keep
a test binary that runs serial and pooled work and then checks that
rayon::ThreadPoolBuilder::new().build_global() still succeeds.
Upgrading from 0.4 or 0.5.0
The leader of a team is pinned onto S[0] for the life of a scope, and for that
one section when a token of a Team dispatches one without opening a scope,
where 0.4.0 left its mask whole. Three things to change:
- a vendor call made by the leader goes through
TeamExec::external, which widens the mask toSfor that call; a direct call inside a share runs on one CPU; Pool::leader_moves()is gone;- the placement belongs to the team: a whole-team scope decides it and records where its members are, while a scope that is not the root token's, or a section dispatched without a scope, pins the leader for that scope or section only, inside the mask the calling thread has.
From 0.5.0 and 0.5.1: both pinned the thread that calls Pool::into_team onto
S[0] until the Team was dropped (pin::TeamPin, still in 0.5.1), so a vendor thread pool
built there (MKL, OpenBLAS), a plain std::thread::spawn and a second Pool all
inherited one CPU. This release (after 0.5.1) does not: that thread keeps its own mask, and the leader
is placed for each section and each scope instead, so a vendor library or a
plain thread started from there is not confined to one CPU.
CHANGELOG.md names every removed item, including the internal
ones, and lists the behaviour that did not change.
Upgrading from 0.1
0.2 still runs entries on a private Rayon pool. The signatures are those of
0.1 plus PoolBuilder::spin_polls / Pool::spin_polls; the behaviour changes
are:
- Pool drop and reuse. A pool with the default thread settings no longer
joins its workers on drop: after its members left, its Rayon pool goes to a
process-wide cache, and the next default pool of the same width reuses those
threads (each worker takes the building thread's CPU set). A pool with a
thread_name, anon_starthook, astack_sizeor aqosof its own still gets dedicated workers that its drop joins. - Waits follow libgomp (
docs/design/waiting.md): a bounded spin ofspin_pollspolls (by default 300000 on x86-64, 2^21 on aarch64), 16 yields, then sleep. A reserved member that sleeps stays reserved; an idle member returns to Rayon afteridle_spinor the spin budget, whichever comes first. - Unchanged:
idle_runs_rayonkeeps its meaning (an idle member serves Rayon jobs while it spins),externalon a team token still lends the members to the pool's Rayon jobs, andparallelstill pulls Rayon.
Wait benchmarks
benches/wait_scenarios.rs measures how team threads wait, the way callers use them; the design
is in docs/design/waiting.md. A change to the waits is judged by running
it on the base and on the head of the change, on a quiet host and on a many-core shared host.
| scenario | a call is | reports |
|---|---|---|
sections |
200 sections of 5, 20 or 50 µs, team held | per call, and the overhead over sections * work |
gaps |
10 sections of 5 µs, the caller sleeps 0 / 100 µs / 1 ms / 10 ms between calls | the first section (wake latency) and the call |
acquire |
50 sections of 20 µs: the team held (a token per call), the pool held (members acquired per call), a new pool per call | per call; for a new pool also the total from build to drop |
nested |
100 splits of the team into pairs, each pair one 20 µs section | per call |
Each scenario runs at width = all CPUs of the affinity set and at three quarters of them, on
threadteam and on a Rayon scope; benches/omp/wait_omp.c runs sections and gaps on libgomp
(one omp parallel region per section, runtime defaults). --load K:P adds K child processes,
each busy P % of every 10 ms, for the whole run: the stand-in for another user's processes.
# one checkout, both passes (quiet, then 8 processes at 10 %), rows appended to
# docs/design/waiting-data/<host>.csv
- evo (quiet):
scripts/bench-waits.sh <label> "" noneon the base and on the head. The acceptance check is thatsectionsandgapsdo not regress. - broccoli (64 CPUs, shared):
taskset -c 0-63 scripts/bench-waits.sh <label> 64,48 8:10on the base and on the head, plus the faer#107 arms ofacquire_probe_107(tensor4all/faer branchprobe/107-acquire-per-call). The acceptance check is that width 64 under load is no longer slow and width 48 does not regress.
To compare two commits on a shared host, scripts/bench-ab.sh <base-rev> <head-rev> <rounds> [bench args] builds both and alternates them, so a drift of other users' load falls on both sides;
HEAD_VARIANTS='--spin-polls 300000;--own-hold-ms 100' adds head runs with tuning flags (this
version only), and OMP=1 runs the libgomp driver in every round with the same scenario, widths,
calls and load (acquire is libgomp's held team). The base of the waits change is its bench commit a3a9320 (the bench on the 0.1.3
waits). Under every threadteam row, a waits per call line counts join and barrier sleeps, member
parks and the preemptions waiters saw.
Results are committed as CSV under docs/design/waiting-data/, one
file per host, with the label naming the commit; the summary tables live in waiting.md.
Resident-split benchmark
benches/resident_split.rs measures what a team fork pays when its shares read data the leader
has just written (issue #27): a copy split over k jobs, disjoint or shared, with the source hot
in the leader's caches or evicted. Run one die at a time and derive the per-host transfer rate
as described in the file header:
License
Licensed under either of MIT or Apache-2.0 at your option.