threadteam 0.6.0

Scoped CPU thread-team placement token: a Pool and a non-Clone Exec with fork, for_ranges, external and broadcast; no Rayon type in the public API.
Documentation

threadteam

Splittable CPU placement token for the tensor4all CPU stack. threadteam is the one place that starts tasks: kernels take an Exec token and split it, and no scheduler type appears in any public signature.

Used by strided-rs, tcontract-rs, cpueinsum-rs, tlinalg-rs and tenferro-rs. Design records: docs/design/placement.md (the token), docs/design/waiting.md (how the workers wait).

The workers belong to a private Rayon pool per Pool. A pool with the default thread settings keeps its Rayon pool in a process-wide cache when it is dropped, and the next pool of the same width takes it, so a pool built per call (faer's with_team) reuses running threads instead of starting new ones.

Usage

use std::sync::atomic::{AtomicUsize, Ordering};
use threadteam::{Exec, Pool};

// The owner builds a pool once and keeps it.
let pool = Pool::builder().threads(4).build()?;

// Per call, the owner mints a token for `budget` slots of the pool.
let mut exec = Exec::new(&pool, 4)?;

// Several phases in sequence: reborrow, because entries consume the token.
let total = AtomicUsize::new(0);
exec.reborrow().for_ranges(1000, 64.try_into().unwrap(), |r| {
    total.fetch_add(r.len(), Ordering::Relaxed);
});

// Independent parts: each job gets a child token with its share of the slots.
exec.fork(2, |_job, child| {
    let _ = child.budget();
});
# Ok::<(), threadteam::Error>(())

Exec::serial() needs no pool and runs everything inline.

Intended consumer pattern

The owner of the CPU resources (an engine, a contraction resource bundle) holds a threadteam::Pool and mints Exec::new(&pool, budget) for each call, passing the token to kernels by value. Kernels never store a token and never mint one.

API

Item Role
Pool, PoolBuilder fixed set of workers; threads, thread_name, stack_size (default 16 MiB, sized for recursive LU), on_start, build
Exec::new(&pool, budget) mint a token; the only clamp (to the pool width); budget == 0 is an error
Exec::serial() the calling thread only; const
Exec::budget, is_serial inspect the token (is_serial means budget 1)
Exec::reborrow(&mut self) same pool and budget while the parent is exclusively borrowed
Exec::limit(max) narrow to min(budget, max)
Exec::fork(jobs, f) min(jobs, budget) tasks; shares differ by at most one and sum to the budget; job i gets a child token
Exec::fork_sized(&shares, f) one child per share, child i gets a token of exactly shares[i] slots; shares sum to at most the budget (Err(Error::SharesExceedBudget) otherwise, never clamped; a zero share is Err(Error::ZeroBudget)); leftover slots stay idle and return to the parent after the join; fork is its even special case
Exec::fork_sized_slots(&shares, &mut slots, f) fork_sized where child i also gets exclusive &mut slots[i] (no lock, so a warm call allocates nothing); each child derives its own &mut slots[i] from its index right before its callback, exclusive because each index is dispatched exactly once, and the borrow of slots ends at the join; a slot count differing from the share count is Err(Error::SlotsMismatch); after a child panic is re-raised every slot holds a valid S the caller can use (the panicking child's slot as it left it)
Exec::for_ranges(len, grain, f) at most budget tasks, ranges of at least grain except possibly the last
Exec::broadcast(f) f(i) for i in 0..budget, all concurrent, or Err(Refused)
Exec::can_broadcast() non-consuming: true exactly when broadcast would not refuse right now; ask before building an expensive setup
Exec::carries_team() true when the token has team authority (minted by a Team or a child of one); team() then returns the carried team without acquisition, so a consumer that wants a TeamExec asks this and keeps external for plain tokens
Exec::team(f) / team_with(over, f) reserve budget - 1 awake pool members for a scope and run barrier-bearing sections through TeamExec (run, run_with, split, fork, reborrow), or Err(Refused::Unavailable); see the design record
Pool::into_team(width) -> Team (owns the pool; IntoTeamError gives it back), Team::exec(budget) / try_exec, Team::pool, Team::into_pool persistent reservation for an engine that owns its pool (Pool is not Clone); minted tokens are ordinary Execs whose fork, for_ranges, broadcast, team dispatch on the reserved members; one leader at a time (exec blocks, try_exec -> Refused::Busy); idle members park while reserved
PoolBuilder::qos(Qos), raise_current_thread_qos(Qos) worker QoS on macOS (default UserInitiated); raise the leading thread without ever lowering it
PoolBuilder::idle_spin, idle_runs_rayon, acquire_timeout; Pool::release_idle waiting policy of idle team members (an idle member returns to Rayon after idle_spin or the spin budget, whichever comes first)
PoolBuilder::spin_polls, Pool::spin_polls spin budget of every team wait, in polls (the shape of libgomp's GOMP_SPINCOUNT): a reserved member waiting for work, the leader in a join and a participant in a barrier spin that many polls, yield a few times, then sleep until woken; throttled to 100 polls when the root scope is oversubscribed (more participants than CPUs in the caller's mask). No wait reads a clock or detects descheduling (docs/design/waiting.md)
PoolBuilder::pin_participants(bool), Pool::pin_participants(), Pool::pin_skips() placement of the participants of a root team scope inside the calling thread's affinity set S (Linux; on by default, false turns it off per pool): the leader is pinned to S[0] in exactly two cases, (1) for the life of a root team scope (Exec::team, or a Team token's team: one sched_setaffinity at entry, one at exit, also on unwind) and (2) for one section when a token of a Team dispatches fork / for_ranges / broadcast without opening a scope (pinned for that section, its own mask back when it ends), and keeps its own mask S everywhere else; member k (participant k >= 1) pins itself onto its own CPU of S other than S[0] (by default round-robin over the NUMA nodes S spans, PinStrategy::Spread; PoolBuilder::pin_strategy(PinStrategy::Pack) gives the 0.4-0.5 placement S[k]), and a member restores its own mask before it returns to Rayon or serves foreign work. Nothing wraps: when S has fewer CPUs than participants nobody is placed and Pool::pin_skips() counts the scope. The members' placement belongs to the team and the leader's pin to one scope, or to one section: the tokens of the thread that called Pool::into_team run fork_sized, for_ranges, ... with every participant on its own CPU without a team scope, and that thread keeps its own mask between sections, so into_team returns without changing it and a thread or a vendor pool started there gets all of S; a token used on another thread places that thread only, for the sections it dispatches, and only inside the mask that thread has. A vendor library creates its thread team from the calling thread and a one-CPU caller would confine the whole provider team, so the leader calls it through TeamExec::external, which sets its mask to S for the call and back to {S[0]} after (two sched_setaffinity calls per call, about 2.3 µs on broccoli, paid every call); a member uses Exec::external (which restores the member's mask first). The mask only says where a provider's threads may run, not how many it starts: that is OMP_NUM_THREADS / OPENBLAS_NUM_THREADS / MKL_NUM_THREADS (see tenferro-rs PERFORMANCE_TIPS.md). The NUMA effect of Spread is not verified on this project's hosts (all one NUMA node). For tensor4all/faer#107 and the evo2 cross-L3 stall; give each team its own CPU set (taskset -c), there is no CPU allocator
DispatchCosts, WidthCosts, PoolBuilder::dispatch_costs, Exec::dispatch_cost(width), TeamExec::dispatch_cost(width) cost of one parallel section per kind of token (team, in pool, outside), steady state; one row per HostClass (HostClass::detect(), keyed by kernel dispatch: x86_64-avx512, x86_64-avx2, x86_64-scalar, aarch64-apple, aarch64, generic), DispatchCosts::for_class(class); measured rows for x86_64-avx512 (team anchors at widths 2, 4, 6 and 8) and aarch64-apple, a measured Zen2 team row for x86_64-avx2 (anchors at ten widths from 2 to 64; its plain kinds stay generic), every other class takes the conservative generic row (3 / 15 / 40 us at the anchors 2, 4, 6); a row holds measured anchors (WidthCosts::new, WidthCosts::flat, or WidthCosts::from_anchors for measured anchors up to 64 and beyond), interpolates between them and grows linearly above the largest one, so a width above the anchors is never charged the cost of the largest anchor; DispatchCosts::host_default() is the detected class's row; Duration::MAX when serial; override per pool
pays_at(est, sections, cost), Exec::pays(est, sections, width), Exec::width_for(est, sections), width_for_by, PAYS the one rule: parallel iff est >= PAYS * sections * D(width); width_for minimises sections * D(k) + est / k
SplitCost, TransferRate, Exec::width_for_split(split), Team::width_for_split(split), width_for_split_by the width model with the transfer of a section whose shares read data the caller has just written: minimises sections * D(k) + est / k + moved(k) / rate with moved(k) = resident_bytes * (k - 1) / k, one drain at one aggregate rate per HostClass (8 ns/KiB measured on aarch64-apple, 41 ns/KiB on x86_64-avx512 and on every unmeasured class); resident_bytes: 0 is exactly width_for, and the transfer is never part of the pay decision (docs/design/thresholds.md §3.8)
Exec::external(f) enter the pool once and hand the budget as a plain width to an outside library (faer, a vendor BLAS); how it divides the width is that library's concern

Contracts

  • No duplication. Exec is neither Clone nor Copy (compile-fail doctests), and every parallel entry consumes it.
  • Slots are conserved. Under one root token the number of tasks running at once is at most its budget at every nesting depth. A parent waiting for its children keeps no slot.
  • Lazy and inline. A serial token, budget one, or one task runs inline on the calling thread and makes no scheduler call, so the process-global Rayon pool is never started implicitly.
  • Placement by caller. A worker of the target pool spawns into the current scope without installing; a plain thread or a worker of another pool enters the target pool once and waits. The crate keeps no ambient thread-local state: the one value it stores per thread is the placement record (pin::PLACED, a non-allocating Cell), and nothing else is inherited by work that runs on the calling thread.
  • broadcast refuses with Refused::ForkChild for a token derived from fork (through reborrow or limit too), and with Refused::CallerInPool when the caller is a worker of the same pool. Roots, and their reborrows and limits, are eligible. Callers must handle Refused by falling back to a barrier-free path. Concurrent broadcasts on one pool queue in the same order on every worker (no interleaving); participants are the pool's workers 0..budget, so blocking work of another token on those workers delays the broadcast. can_broadcast applies the same rule through one private helper, so the two cannot disagree.
  • Shutdown. Dropping a Pool ends its team members and waits until they left. A pool with a thread_name, an on_start hook, a stack_size or a qos of its own has dedicated workers, and its drop joins them: when drop returns, all of them have exited and their thread-local destructors (including state created in an on_start hook) have run. A pool with the default thread settings keeps its idle workers in a process-wide cache keyed by width. The next default pool of that width takes them and gives each worker the building thread's CPU set. A cached set belongs to one pool at a time, and the cache holds at most as many threads as the larger of the available parallelism and the widest pool built so far; it joins the oldest entries beyond that. A pool dropped from one of its own workers cannot wait for itself, so that drop only signals shutdown and returns without joining or caching. Without parallel there is nothing to join.
  • No scheduler type is public. There is no constructor from, and no accessor returning, a scheduler pool. scripts/check-no-public-rayon.sh runs cargo public-api -p threadteam (when installed) and fails if rayon:: appears in the output.

Features

Feature Effect
parallel (default) run on a private pool. Without it the same types exist, PoolBuilder::build returns a width-1 pool and every entry runs inline
ledger test support: ledger::max_live(root), live, installs, and Exec::ledger_root. Records live tasks per root token and pool installs; compiled out when off

Consumers assert ledger::max_live(root) <= budget in their own tests, and keep a test binary that runs serial and pooled work and then checks that rayon::ThreadPoolBuilder::new().build_global() still succeeds.

Upgrading from 0.4 or 0.5.0

The leader of a team is pinned onto S[0] for the life of a scope, and for that one section when a token of a Team dispatches one without opening a scope, where 0.4.0 left its mask whole. Three things to change:

  • a vendor call made by the leader goes through TeamExec::external, which widens the mask to S for that call; a direct call inside a share runs on one CPU;
  • Pool::leader_moves() is gone;
  • the placement belongs to the team: a whole-team scope decides it and records where its members are, while a scope that is not the root token's, or a section dispatched without a scope, pins the leader for that scope or section only, inside the mask the calling thread has.

From 0.5.0 and 0.5.1: both pinned the thread that calls Pool::into_team onto S[0] until the Team was dropped (pin::TeamPin, still in 0.5.1), so a vendor thread pool built there (MKL, OpenBLAS), a plain std::thread::spawn and a second Pool all inherited one CPU. This release (after 0.5.1) does not: that thread keeps its own mask, and the leader is placed for each section and each scope instead, so a vendor library or a plain thread started from there is not confined to one CPU.

CHANGELOG.md names every removed item, including the internal ones, and lists the behaviour that did not change.

Upgrading from 0.1

0.2 still runs entries on a private Rayon pool. The signatures are those of 0.1 plus PoolBuilder::spin_polls / Pool::spin_polls; the behaviour changes are:

  • Pool drop and reuse. A pool with the default thread settings no longer joins its workers on drop: after its members left, its Rayon pool goes to a process-wide cache, and the next default pool of the same width reuses those threads (each worker takes the building thread's CPU set). A pool with a thread_name, an on_start hook, a stack_size or a qos of its own still gets dedicated workers that its drop joins.
  • Waits follow libgomp (docs/design/waiting.md): a bounded spin of spin_polls polls (by default 300000 on x86-64, 2^21 on aarch64), 16 yields, then sleep. A reserved member that sleeps stays reserved; an idle member returns to Rayon after idle_spin or the spin budget, whichever comes first.
  • Unchanged: idle_runs_rayon keeps its meaning (an idle member serves Rayon jobs while it spins), external on a team token still lends the members to the pool's Rayon jobs, and parallel still pulls Rayon.

Wait benchmarks

benches/wait_scenarios.rs measures how team threads wait, the way callers use them; the design is in docs/design/waiting.md. A change to the waits is judged by running it on the base and on the head of the change, on a quiet host and on a many-core shared host.

scenario a call is reports
sections 200 sections of 5, 20 or 50 µs, team held per call, and the overhead over sections * work
gaps 10 sections of 5 µs, the caller sleeps 0 / 100 µs / 1 ms / 10 ms between calls the first section (wake latency) and the call
acquire 50 sections of 20 µs: the team held (a token per call), the pool held (members acquired per call), a new pool per call per call; for a new pool also the total from build to drop
nested 100 splits of the team into pairs, each pair one 20 µs section per call

Each scenario runs at width = all CPUs of the affinity set and at three quarters of them, on threadteam and on a Rayon scope; benches/omp/wait_omp.c runs sections and gaps on libgomp (one omp parallel region per section, runtime defaults). --load K:P adds K child processes, each busy P % of every 10 ms, for the whole run: the stand-in for another user's processes.

# one checkout, both passes (quiet, then 8 processes at 10 %), rows appended to
# docs/design/waiting-data/<host>.csv
scripts/bench-waits.sh <label>                 # e.g. main-4c73c2b, then the PR head
scripts/bench-waits.sh <label> 64,48 8:10      # explicit widths and load
cargo bench --bench wait_scenarios -- --scenario gaps --widths 8   # one scenario; options in the file header
  • evo (quiet): scripts/bench-waits.sh <label> "" none on the base and on the head. The acceptance check is that sections and gaps do not regress.
  • broccoli (64 CPUs, shared): taskset -c 0-63 scripts/bench-waits.sh <label> 64,48 8:10 on the base and on the head, plus the faer#107 arms of acquire_probe_107 (tensor4all/faer branch probe/107-acquire-per-call). The acceptance check is that width 64 under load is no longer slow and width 48 does not regress.

To compare two commits on a shared host, scripts/bench-ab.sh <base-rev> <head-rev> <rounds> [bench args] builds both and alternates them, so a drift of other users' load falls on both sides; HEAD_VARIANTS='--spin-polls 300000;--own-hold-ms 100' adds head runs with tuning flags (this version only), and OMP=1 runs the libgomp driver in every round with the same scenario, widths, calls and load (acquire is libgomp's held team). The base of the waits change is its bench commit a3a9320 (the bench on the 0.1.3 waits). Under every threadteam row, a waits per call line counts join and barrier sleeps, member parks and the preemptions waiters saw.

Results are committed as CSV under docs/design/waiting-data/, one file per host, with the label naming the commit; the summary tables live in waiting.md.

Resident-split benchmark

benches/resident_split.rs measures what a team fork pays when its shares read data the leader has just written (issue #27): a copy split over k jobs, disjoint or shared, with the source hot in the leader's caches or evicted. Run one die at a time and derive the per-host transfer rate as described in the file header:

numactl --cpunodebind=0 --membind=0 cargo bench --bench resident_split -- --widths 1,2,4,8 > out.csv

License

Licensed under either of MIT or Apache-2.0 at your option.