1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
//! Per-worker TSC-event trace facility for one-shot scheduler debugging.
//!
//! Each Flynnel worker (and the external caller thread) accumulates
//! `(event_kind, payload, tsc)` tuples into a thread-local buffer at
//! pre-defined hook points. After a dispatch completes,
//! [`dump_to_stderr`] serializes every recorded buffer to stderr as
//! a CSV stream (one row per event) so the timing of the dispatch
//! can be reconstructed offline.
//!
//! # Enabling
//!
//! Off by default. Set environment variable `FLYNNEL_TRACE=1` before
//! the process starts, or call [`set_enabled`] once it is running.
//! The first [`is_enabled`] call seeds the flag from the variable;
//! every call after that is a `Once` guard and one relaxed load.
//!
//! # Cost when off
//!
//! Single Relaxed atomic load + branch (`if !is_enabled() { return; }`).
//! Per recorded event when on: one `read_tsc()` (~20 cycles) + one
//! `Vec::push` on a thread-local buffer (amortized O(1)). The
//! thread-local sits behind `RefCell` so the borrow is cheap and
//! single-threaded; no atomics on the recording path.
//!
//! # Why a binary (not the bench harness)
//!
//! Criterion runs hundreds of iterations per measurement; tracing
//! every iteration produces gigabytes of CSV. The companion
//! `examples/trace_dispatch.rs` binary traces one dispatch with
//! `FLYNNEL_TRACE=1`, dumps every thread's buffer, then exits.
use RefCell;
use ;
/// Event kinds emitted at the instrumented hook points. Compact `u8`
/// so the per-event memory cost stays low.
/// One trace event row recorded into the per-thread buffer.
thread_local!
/// Process-wide enable flag, seeded from `FLYNNEL_TRACE` on the first
/// call to [`is_enabled`] and settable afterward.
///
/// An `AtomicBool` rather than a `OnceLock<bool>` so the ring can be
/// turned on after the process has started. Latched, nothing could:
/// a caller who did not set the variable before launch had no way to
/// trace at all, and a module cannot set the environment of a process
/// it is already inside.
///
/// The cost of that is one atomic load on a path consulted twice per
/// leaf, on top of the `Once` guard that the latch also paid.
///
/// Measured on the Zen 3 guest with `examples/trace_predicate_cost`,
/// which times both shapes interleaved in one process against a
/// plain-bool control, five hundred million calls a cell, median of
/// seven. Two readable runs put the difference at +0.0059 and
/// -0.0350 ns a call: it straddles zero, so the change is not
/// resolvable and is bounded at 0.014 percent of a 200,000-element
/// dispatch. Both shapes sit about 0.08 ns above the control, which
/// is under a cycle, so the pipeline absorbs either one.
static TRACE_ENABLED: AtomicBool = new;
/// Reads `FLYNNEL_TRACE` once, before the flag is first answered.
static TRACE_FROM_ENV: Once = new;
/// Strictly-monotonic id for `register_thread`, so the dump can
/// associate each recorded buffer with a thread name.
static THREAD_ID_GEN: AtomicU64 = new;
/// Process-wide flag that the worker_loop reads at the top of each
/// iteration. When set, the worker flushes its trace buffer to
/// stderr (tagged with `flynnel-worker-{idx}`) and clears its
/// thread-local. Use [`request_worker_flush`] to set; the workers
/// auto-clear after dumping so consecutive flushes work.
static WORKER_FLUSH_FLAG: AtomicBool =
new;
/// Signal all worker_loop iterations to dump their trace buffers
/// on the next loop pass. Workers check the flag at the top of
/// each iteration and dump+clear if set. The flag is consumed
/// per-thread via a thread-local last-seen counter so all workers
/// observe each flush request exactly once.
thread_local!
/// Worker_loop hook: check whether a flush has been requested. If
/// yes and this thread hasn't already handled this request, dump
/// the trace buffer (tagged with `label`) and mark this thread
/// done. Returns true if a dump happened.
/// Count of worker dumps completed since process start. A worker's
/// dump writes every row it holds through the stderr lock, which
/// takes milliseconds per worker; a requester waits on this count
/// before exiting, or the newest rows are cut off.
static FLUSHES_DONE: AtomicU64 = new;
/// The number of worker dumps that have completed in full.
/// Caller-side: reset the request after a dump cycle so subsequent
/// requests can fire.
/// Whether the per-thread trace ring is recording.
///
/// Seeded from `FLYNNEL_TRACE` on the first call and settable after
/// that with [`set_enabled`].
/// Seeds the flag from the environment, once.
///
/// Separate from the load so the env read happens exactly once and
/// the steady-state cost is the `Once` guard plus one relaxed load.
/// Turns the trace ring on or off, answering what it held before.
///
/// Seeds from the environment first, so a caller who turns it off has
/// turned off whatever the variable asked for rather than racing the
/// seeding.
///
/// Recording is per thread and the rings are drained by
/// `dump_to_stderr` or a worker flush, so turning it on mid-process
/// records from the next event on each thread and says nothing about
/// what happened before.
/// Record one event on the calling thread's trace buffer. Cheap
/// fast-exit when tracing is disabled.
/// Reset all thread buffers known to this thread (and our own).
/// Call before a dispatch you want to trace cleanly.
/// A copy of every event the calling thread's ring holds, oldest
/// first.
///
/// The ring is left as it is; [`reset_current_thread`] is what empties
/// it. Only the calling thread's ring can be read, because each ring is
/// a thread-local: a worker's events reach stderr through
/// [`request_worker_flush`] and nowhere else.
///
/// A ring records only while [`is_enabled`] answers true and keeps
/// what it recorded after tracing is turned off, so an empty answer is
/// a thread that recorded nothing since its ring was last emptied.
/// Reading a thread's ring for the first time allocates it, as its
/// first event would.
/// Per-thread registration. Worker threads call this once at startup
/// so the dump knows who they are. Returns an integer id used in
/// the dumped CSV.
/// Dump the calling thread's trace buffer to stderr as CSV. Other workers
/// have to flush themselves before the dump can include them; the
/// worker loop has a "trace dump" path that writes its buffer on
/// demand.
///
/// CSV columns: thread_name,event_kind_num,payload,tsc
/// Worker-loop hook: each worker's loop calls this after it runs a
/// job so the buffer is flushed at a known quiescent point. The
/// `name` is `flynnel-worker-{idx}` typically.
///
/// `examples/trace_dispatch.rs` instead requests a flush with
/// [`request_worker_flush`] and waits on [`worker_flushes_done`].