1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
//! How many threads one link runs on.
//!
//! Every parallel pass fans out to rayon's global pool, which is one worker
//! per core. On a small link that pool costs far more than it earns: the
//! workers spin in the work-stealing loop with nothing to steal, which is
//! kernel time spent against no user work. A compiler driver's link of one
//! translation unit runs about 1.45 times slower in wall clock on the global
//! pool than on one thread, for 2.5 times the CPU.
//!
//! So a link sizes its own pool once, up front, from the workload it can see
//! after the inputs are open. Below the threshold it runs on a private
//! one-thread pool and the global pool is never built at all; above it nothing
//! is installed and the link uses the global pool exactly as before. Both
//! paths run the same code over the same data, so the image is the same bytes
//! either way; `tests/pool_threshold.rs` pins that.
use ;
use ThreadPoolBuilder;
use crateET_DYN;
/// The variable rayon reads for the size of its global pool.
///
/// A caller who sets it has already answered this question, so the rule below
/// applies only when it is absent.
const THREAD_VAR: &str = "RAYON_NUM_THREADS";
/// What one input file is worth, in bytes of content, when estimating work.
///
/// A link's cost has a per-byte part (reading sections, copying them,
/// applying relocations) and a per-file part (opening, parsing the header,
/// walking the section table). Fitted timings put the per-file part at about
/// 26 us against about 3.8 us per KiB, so one input is worth roughly 6 KiB.
const FILE_WORK: u64 = 6 * 1024;
/// The estimated work at which the global pool starts paying for itself.
///
/// Below this the fan-out loses wall clock outright; above it the win grows
/// with the workload. Hosts disagree about where the crossover lies, so this
/// is the midpoint of the interval measurement leaves open rather than any
/// one machine's crossover.
const WORK_THRESHOLD: u64 = 768 * 1024;
/// The half-word an ELF header keeps `e_type` in. xold reads little-endian
/// ELF only, so it can be taken straight out of the mapped bytes without
/// parsing the file.
const E_TYPE: = 16..18;
/// The bytes of `input` that become link work.
///
/// A shared object contributes none: it is a `DT_NEEDED` dependency, so the
/// linker reads its dynamic symbol tables and never touches its sections. The
/// distinction is load-bearing, not a refinement -- a driver's link names a
/// 2.4 MiB `libc.so.6` against about 7 KiB of real input, so counting it puts
/// the very link this threshold exists for on the wrong side.
/// Whether `inputs` are worth fanning out over.
///
/// The estimate is the total [`content_bytes`] plus [`FILE_WORK`] per input,
/// compared against [`WORK_THRESHOLD`]. Both terms are needed; neither
/// predicts the crossover on its own. A count gate alone is wrong for a
/// single `ld -r` object carrying eighteen thousand sections, which the pool
/// links faster with no cross-file fan-out at all, since the parallel passes
/// iterate over input sections rather than files. A byte gate alone is wrong
/// for an object that is mostly section header table, which is cheap to walk
/// and prefers one thread despite outweighing workloads that must fan out.
///
/// Counting an archive's whole size is deliberate even though a link pulls
/// only some members: building the archive index and resolving against it is
/// itself work that fans out.
///
/// # Re-fitting the constants
///
/// The crossover moves with core count, memory bandwidth and allocator, and
/// two hosts already disagree about where it is by a third. Re-measure every
/// input shape rather than one: a sweep over object count alone finds a
/// threshold wrong by a factor of three for the others. The requirement is
/// that no link runs slower than it did with the pool always on, so compare
/// the rule-chosen wall clock against a forced multi-thread run at every
/// fixture size, and read CPU alongside it -- a row that is a wash in wall
/// clock but far worse in CPU is still worth taking serially, because under
/// `make -j` that CPU comes out of a concurrent compile.
/// The estimated work in `inputs`: their [`content_bytes`] plus
/// [`FILE_WORK`] each. Two decisions read it, so it is computed once.
/// The estimated work above which transparent huge pages stop paying.
///
/// mimalloc reserves a 1 GiB arena and marks all of it `MADV_HUGEPAGE`, so
/// every anonymous allocation a link makes faults as a 2 MiB page. Two costs
/// follow. A huge page is zeroed in full on first touch, and a link writes
/// its arena sparsely and once -- it streams through its inputs rather than
/// revisiting a working set -- so the TLB reuse that pays for huge pages
/// elsewhere never accrues. And under Linux's default `defrag=madvise` a
/// fault with no free 2 MiB block compacts *synchronously* in the faulting
/// thread, which is a stall no other thread overlaps. On the host this was
/// fitted on, 92% of the kernel's compaction attempts were already failing.
///
/// # What this was fitted to
///
/// 600 independent objects of about 0.77 MB each, linked as a dynamic
/// executable in prefixes; medians of 11 interleaved pairs per row, with
/// memory compacted before each row so no row inherits the previous one's
/// fragmentation. The ratio is huge-pages-off over huge-pages-on, so below
/// 1.0 turning them off wins.
///
/// | objects | estimate | ratio | disabled |
/// |---|---|---|---|
/// | 120 | 96 MB | 1.185 | no |
/// | 200 | 160 MB | 0.702 | no |
/// | 280 | 224 MB | 0.723 | no |
/// | 360 | 288 MB | 0.705 | yes |
/// | 440 | 353 MB | 0.680 | yes |
/// | 520 | 417 MB | 0.650 | yes |
/// | 600 | 481 MB | 0.602 | yes |
///
/// The real corpora agree at both ends: the 2864-object LLVM link (389 MB
/// estimated) measured 0.68x to 0.79x across four sessions and never once
/// preferred huge pages, while a 121-object C corpus (51 MB) measured 1.115x
/// and the same corpus against `libc.a` (81 MB) 1.091x.
///
/// # Why not at the crossover
///
/// The sweep puts the crossover between 96 MB and 160 MB, but two real
/// corpora sitting at 229 MB and 238 MB -- the same C++ objects linked shared
/// and dynamic -- changed sign between sessions, measuring 1.05x once and
/// 0.86x another time. Most of their bulk is DWARF, which is copied from one
/// mapping to another rather than held on the heap, so the byte estimate
/// overstates the anonymous footprint that actually faults. The threshold
/// therefore sits above them rather than at the crossover: every shape at or
/// above 256 MB won in every session it was measured in, and no shape below
/// it is put at risk. The cost of the choice is the 160 MB and 224 MB rows,
/// which give up a measured 0.70x.
///
/// # If this is re-fitted
///
/// The crossover moves with how fragmented the host's memory is, which is
/// what makes it worth a threshold rather than an unconditional call: on a
/// long-lived, fragmented machine turning huge pages off won every row of
/// the sweep, including the ones that prefer them here. Compact memory
/// (`/proc/sys/vm/compact_memory`) before each row, or the measurement reads
/// the previous row's fragmentation rather than the workload.
const HUGE_PAGE_THRESHOLD: u64 = 256 * 1024 * 1024;
/// Whether the process has allowed its huge-page policy to be changed.
///
/// Off unless [`allow_huge_page_tuning`] is called. The policy is
/// process-wide and permanent, which is the application's call to make and
/// not a library's: a host that links once and then runs for hours would
/// otherwise lose huge pages everywhere on the strength of that one link.
static TUNE_HUGE_PAGES: AtomicBool = new;
/// Lets a link turn transparent huge pages off when the workload is large
/// enough to be hurt by them. See [`HUGE_PAGE_THRESHOLD`].
///
/// Intended for the `xold` binary, whose process exists only to link.
/// Opts the process out of transparent huge pages, if it allowed it.
/// Runs `job` on a thread pool sized for `inputs`.
///
/// Three cases leave the ambient pool alone: an explicit [`THREAD_VAR`], a
/// pool the caller installed around this call, and a workload [`fans_out`]
/// judges big enough. Only a small link in an unconfigured process gets a
/// private one-thread pool.
pub