1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
//! Participant placement on the calling thread's CPU set (on by default).
//!
//! With [`PoolBuilder::pin_participants`](crate::PoolBuilder::pin_participants)
//! on (the default), the leader of a root team scope reads its own affinity mask
//! `S` and places the team inside it, never outside it:
//!
//! * **The leader is pinned to `S[0]` for the life of the scope**, and the
//! thread that created a `Team` for the life of the team ([`TeamPin`]),
//! which is what places a team token that never opens a scope. One
//! `sched_setaffinity` at scope entry ([`LeaderPin::enter`]) and one at scope
//! exit (its drop, which also runs on unwind) restore `S`; there is no check
//! and no move before a section. The price is that a vendor library creates
//! its thread team from the thread that calls it, and a team created from a
//! one-CPU thread silently confines the whole library team to that CPU
//! (tenferro-rs `PERFORMANCE_TIPS.md`). The leader's half of that rule is
//! [`widened`], behind `TeamExec::external`: it sets the mask to `S` for the
//! duration of one call and puts `{S[0]}` back when the call returns or
//! unwinds.
//! * **Member `k >= 1` pins itself onto `S[k]`**, so the members take `S` minus
//! `S[0]`, one CPU each. A member saves its previous mask in its own slot and
//! restores it itself before it stops being reserved.
//! * **No wrapping.** When `S` has fewer CPUs than the team has participants
//! nothing is pinned (today's unpinned placement), the members of a persistent
//! team that an earlier scope pinned restore their own masks, and the skip is
//! counted ([`Pool::pin_skips`](crate::Pool::pin_skips)). Wrapping would stack
//! two members on one CPU, which is worse than leaving them to the scheduler.
//!
//! Every affinity change is made by the thread that is being moved, with
//! `sched_setaffinity(0, ...)`, so no thread ever changes another thread's
//! placement and there is no thread-id bookkeeping. Linux only: on other
//! targets the option is accepted and does nothing. Nothing here allocates: the
//! masks are `cpu_set_t` values on the stack and the `k`-th set CPU is found by
//! scanning one.
//!
//! The same CPU set decides whether a root scope is *oversubscribed* (see
//! [`available`]): a team of more participants than the caller's set has CPUs
//! parks its members early instead of spinning (see `awake.rs`), and
//! [`oversubscribed`] is what the root-scope entries use. On a leader this
//! crate placed, the caller's set is the `S` it was placed in, not the one-CPU
//! mask it now runs on. An oversubscribed root scope is one whose pin is
//! skipped; the converse fails only for a root scope of another pool entered on
//! a placed leader, which reads the one-CPU mask, places nobody, and is paced
//! by `S`.
use ;
pub use count;
pub use count;
pub use ;
/// Number of CPUs the calling thread's team is spread over: the CPUs a root
/// team scope entered on this thread may place its participants on.
///
/// This is *not* "CPUs in the calling thread's current affinity mask" once
/// this crate has placed the thread. A placed leader (a [`LeaderPin`], which a
/// [`TeamPin`] holds for the life of a `Team`) runs on the one CPU `{S[0]}`,
/// but it still leads a team spread over `S`, so the answer is `|S|`, read
/// from the placement this thread records. Answering 1 there marked every
/// token a `Team` mints on its own leader thread as oversubscribed and
/// throttled its members' waits (threadteam-rs#33).
///
/// On any other Linux thread it is the number of CPUs in its affinity mask,
/// i.e. the same set the pin section places participants on. Elsewhere
/// [`std::thread::available_parallelism`]. Zero is never returned.
pub
/// Whether a root scope with `participants` participants is oversubscribed: it
/// has more participants than the calling thread's team has CPUs
/// ([`available`]: `|S|` on a placed leader, its mask elsewhere).
///
/// Evaluated on the thread that enters the root scope, whether or not pinning
/// is on. For a [`Team`](crate::Team) that is the thread that mints the root
/// token *and* again the thread that opens each team scope through it, so a
/// token that travels to a thread with a different CPU set is paced by that
/// thread's set.
pub
/// Pins the calling thread onto the `k`-th CPU of `set`.
///
/// Called by a member for itself (`k >= 1`); the leader never calls it. Never
/// wraps: `k >= |set|` leaves the thread where it was (the root scope does not
/// pin at all in that case, see `exec::pin_scope`, so this is only a second
/// line of defence). It does not look at the current mask first: it asks the
/// kernel for the single CPU `set[k]` wherever the thread happens to be, so a
/// thread that was outside `set` is moved into it. Best effort: an empty set,
/// or a refused syscall (the CPU is not in the caller's permitted cpuset),
/// leaves the thread where it was.
pub
/// What a placed root scope remembers about its leader thread: the whole set
/// `S` it started with and the one CPU, `S[0]`, it is pinned to.
thread_local!
/// The calling thread, the leader of a placed root scope, pinned to `S[0]`.
///
/// Dropping it gives the thread `S` back, on the normal path and on unwind.
/// Nested scopes on one thread stack: the previous placement is put back.
pub
/// The thread that created a [`Team`](crate::Team) (`Pool::into_team*`), its
/// leader, pinned to `S[0]` for the life of the team.
///
/// A root scope's pin ends with the scope, and `into_team` opens one only to
/// acknowledge, so a team driven by tokens that never open a scope
/// (`fork_sized`, `for_ranges`, `broadcast`) would dispatch with its members on
/// `S[1..]` and its leader on its whole mask. The team holds this pin instead:
/// one `sched_setaffinity` when the team is created, one when it is dropped or
/// turned back into its pool, nothing per token or per section. A root scope
/// opened on this thread runs under this pin and places the members in this
/// `S` instead of reading the (now one-CPU) mask. `TeamExec::external` is the
/// only call that widens it, for the duration of one call.
///
/// The pin belongs to the creating thread. A `Team` is `Send` and `Sync`: a
/// token minted on another thread dispatches from a leader that is not placed,
/// and a team dropped on another thread restores nothing (that thread was not
/// moved), so the creating thread keeps `{S[0]}`.
pub
/// Runs `f` with the calling thread's mask widened to `S` when it leads a
/// placed root scope, and puts `{S[0]}` back when `f` returns or unwinds. On
/// any other thread, or in a scope that placed nobody, it just calls `f`.
///
/// Two `sched_setaffinity` calls per call, paid every time: nothing caches
/// "already wide" and nothing skips the restore. A refused widening leaves the
/// thread as it was (on `{S[0]}`) and `f` still runs, then no restore is
/// needed; a refused restore leaves the thread wide until the scope exits and
/// [`LeaderPin`] gives `S` back.
pub
/// The mask a member saved before it pinned itself, and whether it is pinned.
///
/// Written and read only by the thread that owns the slot: a slot id `i` is
/// only ever entered by the worker with index `i` (`claim_job` uses
/// `rayon::current_thread_index()`), and only that thread touches this value —
/// from its `member_loop`, and from `Awake::unpin_member` when the same thread
/// is about to run work that is not its team's (`Exec::external`). The leader
/// never reads it, which is why one saved mask per slot and no
/// cross-reservation cache is enough.
pub
// SAFETY: the `UnsafeCell` is only reached through `save_if_unpinned` and
// `restore`, and both are called exclusively from threads that own the slot: a
// pool worker with index `i` is one fixed thread, it runs the member loop of
// slot `i` only, and `Awake::unpin_member(i)` (`sched::unpin_worker`) is called
// on that same worker. The state those calls leave behind (a saved mask,
// `pinned`) is restored before the loop returns the thread to Rayon, so no
// other thread observes it. `pinned` itself is an atomic, so even a spurious
// read from elsewhere would be sound.
unsafe
// the option is accepted and does nothing off Linux