flynnel 0.6.0

K-aware NUMA-aware work-stealing scheduler with extended-Flynn-taxonomy dispatch (MIMD / SIMC / MIMC / MISD / SIMT / MIMT): a rayon-shaped pool reshaped around per-call execution-class plans with first-class CPU+GPU hybrid (join_hybrid, hybrid_pipeline) and per-call dispatch profiles.
Documentation
//! Per-worker TSC-event trace facility for one-shot scheduler debugging.
//!
//! Each Flynnel worker (and the external caller thread) accumulates
//! `(event_kind, payload, tsc)` tuples into a thread-local buffer at
//! pre-defined hook points. After a dispatch completes,
//! [`dump_to_stderr`] serializes every recorded buffer to stderr as
//! a CSV stream (one row per event) so the timing of the dispatch
//! can be reconstructed offline.
//!
//! # Enabling
//!
//! Off by default. Set environment variable `FLYNNEL_TRACE=1` before
//! the process starts, or call [`set_enabled`] once it is running.
//! The first [`is_enabled`] call seeds the flag from the variable;
//! every call after that is a `Once` guard and one relaxed load.
//!
//! # Cost when off
//!
//! Single Relaxed atomic load + branch (`if !is_enabled() { return; }`).
//! Per recorded event when on: one `read_tsc()` (~20 cycles) + one
//! `Vec::push` on a thread-local buffer (amortized O(1)). The
//! thread-local sits behind `RefCell` so the borrow is cheap and
//! single-threaded; no atomics on the recording path.
//!
//! # Why a binary (not the bench harness)
//!
//! Criterion runs hundreds of iterations per measurement; tracing
//! every iteration produces gigabytes of CSV. The companion
//! `examples/trace_dispatch.rs` binary traces one dispatch with
//! `FLYNNEL_TRACE=1`, dumps every thread's buffer, then exits.

use core::cell::RefCell;
use core::sync::atomic::{AtomicBool, AtomicU64, Ordering};

/// Event kinds emitted at the instrumented hook points. Compact `u8`
/// so the per-event memory cost stays low.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(u8)]
pub enum TraceEvent {
    /// `for_each_chunk` entry. Payload = total item count.
    DispatchEnter = 1,
    /// `for_each_chunk` return. Payload = total item count.
    DispatchExit = 2,
    /// A leaf body is about to execute. Payload = 0.
    LeafStart = 3,
    /// The leaf body completed. Payload = 0.
    LeafEnd = 4,
    /// `join_in_worker` pushed the right half to the deque, with
    /// payload 0, or a cooperative fan-out published its jobs, with
    /// payload = its closure count.
    JoinPush = 5,
    /// `join_in_worker` began its drain loop waiting for the right
    /// half to finish, with payload 0, or a cooperative fan-out began
    /// waiting on its siblings, with payload = its closure count.
    JoinWaitBegin = 6,
    /// The wait ended. From `join_in_worker`, payload = 1 when a thief
    /// had already run the right half and set its latch, and 0 when the
    /// joining thread found the right half still in its deque and ran
    /// it itself. From a cooperative fan-out, payload = its closure
    /// count.
    JoinWaitEnd = 7,
    /// A sampled park ended in a wake. Payload = what the wake cost in
    /// nanoseconds, from the unparker's stamp to this thread running,
    /// saturating at `u32::MAX`. Only the parks the wait controller
    /// samples emit it; an unsampled park, or one that ended without
    /// an unpark, emits none.
    WorkerWake = 8,
    /// A thief stole from a peer. No path in the library emits it.
    StealHit = 9,
    /// An external caller pushed its wrapped join to a slot deque
    /// and broadcast-woke the primaries. Payload = 0.
    SlotPush = 10,
    /// The external caller's wait on the slot job ended. Payload =
    /// 1 when it ended while spinning, 0 after parking.
    SlotWaitEnd = 11,
    /// A primary began running an external caller's wrapped join.
    /// Payload = 0.
    SlotJobStart = 12,
    /// The wrapped join returned on the primary. Payload = 0.
    SlotJobEnd = 13,
    /// A parker is about to wait. Payload = the strategy it chose,
    /// 0 kernel park, 1 WAITPKG, 2 MONITORX, plus 16 when this park
    /// is one half of a controller probe and is therefore being
    /// timed.
    ///
    /// The strategy is recorded per park rather than read from the
    /// controller afterward, because the controller reports where it
    /// ended up and a run that switched partway through looks from
    /// its report exactly like one that started there.
    ParkEnter = 14,
    /// The wait controller moved the process to a different strategy.
    /// Payload = the strategy now in use, same encoding as
    /// [`TraceEvent::ParkEnter`].
    ///
    /// The report says how many times this happened; this says when,
    /// which is what tells a reader whether an early measurement was
    /// taken before or after the move.
    WaitSwitch = 15,
    /// A pool worker committed to parking in the JEC sleep coordinator.
    /// Payload = its worker index. Distinct from [`TraceEvent::ParkEnter`],
    /// which is the latch parker a waiting caller uses.
    PoolPark = 16,
    /// A producer claimed a parked pool worker to wake it. Payload =
    /// the claimed worker's index. Distinct from
    /// [`TraceEvent::WorkerWake`], which the latch parker emits on its
    /// own return.
    PoolWake = 17,
    /// A pool worker began an idle episode whose spin rounds wait on
    /// the host's monitor rather than yield. Payload = 1 for WAITPKG,
    /// 2 for MONITORX. Emitted once per episode, on its first round,
    /// so a traced dispatch counts episodes and not rounds.
    SpinMonitor = 18,
    /// A caller waiting on a latch took the host's monitor for part of
    /// its spin. Payload = 1 for WAITPKG, 2 for MONITORX, plus 32 when
    /// the wait was the slot path's rather than a LockLatch's.
    /// Emitted once per wait, on its first monitor round, so a traced
    /// dispatch counts waits and not rounds.
    ///
    /// A run whose latch-monitor arm records none of these did not
    /// engage, whatever its timings say, and the two cases are
    /// otherwise identical in the output.
    LatchMonitor = 19,
    /// A thief is about to set the latch of a stolen job it has just
    /// run. Payload = 0. The joining thread may observe the latch at
    /// any time after this row.
    LatchSet = 20,
    /// A join waiter's wait ended after at least one yield. Payload =
    /// the last yield's length in microseconds, saturating at
    /// `u32::MAX`. Emitted just before the wait's
    /// [`TraceEvent::JoinWaitEnd`], so a last yield that ends long after
    /// the awaited job's [`TraceEvent::LatchSet`] is a waiter that was
    /// off its core while its latch was set.
    JoinLastYield = 21,
}

/// One trace event row recorded into the per-thread buffer.
#[derive(Copy, Clone, Debug)]
pub struct TraceRecord {
    /// Which instrumentation point emitted this row.
    pub event: TraceEvent,
    /// Per-event integer payload (item count, worker id, etc.).
    pub payload: u32,
    /// Raw TSC value captured at emission time.
    pub tsc: u64,
}

thread_local! {
    /// Per-thread trace buffer. Pre-grown to 16k slots so the first
    /// few thousand events do not pay alloc cost. Buffer is fully
    /// owned by this thread; no atomics on the hot path.
    static THREAD_TRACE: RefCell<Vec<TraceRecord>> =
        RefCell::new(Vec::with_capacity(16_384));
}

/// Process-wide enable flag, seeded from `FLYNNEL_TRACE` on the first
/// call to [`is_enabled`] and settable afterward.
///
/// An `AtomicBool` rather than a `OnceLock<bool>` so the ring can be
/// turned on after the process has started. Latched, nothing could:
/// a caller who did not set the variable before launch had no way to
/// trace at all, and a module cannot set the environment of a process
/// it is already inside.
///
/// The cost of that is one atomic load on a path consulted twice per
/// leaf, on top of the `Once` guard that the latch also paid.
///
/// Measured on the Zen 3 guest with `examples/trace_predicate_cost`,
/// which times both shapes interleaved in one process against a
/// plain-bool control, five hundred million calls a cell, median of
/// seven. Two readable runs put the difference at +0.0059 and
/// -0.0350 ns a call: it straddles zero, so the change is not
/// resolvable and is bounded at 0.014 percent of a 200,000-element
/// dispatch. Both shapes sit about 0.08 ns above the control, which
/// is under a cycle, so the pipeline absorbs either one.
static TRACE_ENABLED: AtomicBool = AtomicBool::new(false);

/// Reads `FLYNNEL_TRACE` once, before the flag is first answered.
static TRACE_FROM_ENV: std::sync::Once = std::sync::Once::new();

/// Strictly-monotonic id for `register_thread`, so the dump can
/// associate each recorded buffer with a thread name.
static THREAD_ID_GEN: AtomicU64 = AtomicU64::new(0);

/// Process-wide flag that the worker_loop reads at the top of each
/// iteration. When set, the worker flushes its trace buffer to
/// stderr (tagged with `flynnel-worker-{idx}`) and clears its
/// thread-local. Use [`request_worker_flush`] to set; the workers
/// auto-clear after dumping so consecutive flushes work.
static WORKER_FLUSH_FLAG: core::sync::atomic::AtomicBool =
    core::sync::atomic::AtomicBool::new(false);

/// Signal all worker_loop iterations to dump their trace buffers
/// on the next loop pass. Workers check the flag at the top of
/// each iteration and dump+clear if set. The flag is consumed
/// per-thread via a thread-local last-seen counter so all workers
/// observe each flush request exactly once.
pub fn request_worker_flush() {
    WORKER_FLUSH_FLAG.store(true, core::sync::atomic::Ordering::Release);
}

thread_local! {
    /// Whether this worker thread has handled the current flush
    /// request. Reset whenever a fresh request arrives (i.e. when
    /// the global flag transitions to true while ours is also
    /// true, we re-flush).
    static FLUSH_OBSERVED: core::cell::Cell<bool> = const { core::cell::Cell::new(false) };
}

/// Worker_loop hook: check whether a flush has been requested. If
/// yes and this thread hasn't already handled this request, dump
/// the trace buffer (tagged with `label`) and mark this thread
/// done. Returns true if a dump happened.
pub fn worker_loop_maybe_flush(label: &str) -> bool {
    if !is_enabled() {
        return false;
    }
    let requested = WORKER_FLUSH_FLAG.load(core::sync::atomic::Ordering::Acquire);
    let already_done = FLUSH_OBSERVED.with(|c| c.get());
    if requested && !already_done {
        dump_to_stderr(label);
        FLUSH_OBSERVED.with(|c| c.set(true));
        FLUSHES_DONE.fetch_add(1, Ordering::Release);
        return true;
    }
    if !requested && already_done {
        // Caller cleared the request; reset our per-thread state
        // so the next request can fire.
        FLUSH_OBSERVED.with(|c| c.set(false));
    }
    false
}

/// Count of worker dumps completed since process start. A worker's
/// dump writes every row it holds through the stderr lock, which
/// takes milliseconds per worker; a requester waits on this count
/// before exiting, or the newest rows are cut off.
static FLUSHES_DONE: AtomicU64 = AtomicU64::new(0);

/// The number of worker dumps that have completed in full.
pub fn worker_flushes_done() -> u64 {
    FLUSHES_DONE.load(Ordering::Acquire)
}

/// Caller-side: reset the request after a dump cycle so subsequent
/// requests can fire.
pub fn clear_worker_flush_request() {
    WORKER_FLUSH_FLAG.store(false, core::sync::atomic::Ordering::Release);
}

/// Whether the per-thread trace ring is recording.
///
/// Seeded from `FLYNNEL_TRACE` on the first call and settable after
/// that with [`set_enabled`].
#[inline]
pub fn is_enabled() -> bool {
    trace_from_env();
    TRACE_ENABLED.load(Ordering::Relaxed)
}

/// Seeds the flag from the environment, once.
///
/// Separate from the load so the env read happens exactly once and
/// the steady-state cost is the `Once` guard plus one relaxed load.
#[inline]
fn trace_from_env() {
    TRACE_FROM_ENV.call_once(|| {
        let on = std::env::var("FLYNNEL_TRACE")
            .map(|v| v == "1" || v.eq_ignore_ascii_case("on") || v.eq_ignore_ascii_case("true"))
            .unwrap_or(false);
        TRACE_ENABLED.store(on, Ordering::Relaxed);
    });
}

/// Turns the trace ring on or off, answering what it held before.
///
/// Seeds from the environment first, so a caller who turns it off has
/// turned off whatever the variable asked for rather than racing the
/// seeding.
///
/// Recording is per thread and the rings are drained by
/// `dump_to_stderr` or a worker flush, so turning it on mid-process
/// records from the next event on each thread and says nothing about
/// what happened before.
pub fn set_enabled(on: bool) -> bool {
    trace_from_env();
    TRACE_ENABLED.swap(on, Ordering::Relaxed)
}

#[inline]
fn read_tsc_local() -> u64 {
    #[cfg(target_arch = "x86_64")]
    // SAFETY: `_rdtsc` is part of the base x86_64 ISA (Pentium and
    // newer); the intrinsic has no CPU-feature preconditions and
    // no architectural preconditions on operand state. The
    // `#[cfg(target_arch = "x86_64")]` guard above is the only
    // gate the call site needs to be sound.
    unsafe {
        std::arch::x86_64::_rdtsc()
    }
    #[cfg(not(target_arch = "x86_64"))]
    {
        std::time::Instant::now().elapsed().as_nanos() as u64
    }
}

/// Record one event on the calling thread's trace buffer. Cheap
/// fast-exit when tracing is disabled.
#[inline]
pub fn emit(event: TraceEvent, payload: u32) {
    if !is_enabled() {
        return;
    }
    let tsc = read_tsc_local();
    let rec = TraceRecord { event, payload, tsc };
    THREAD_TRACE.with(|buf| {
        let mut v = buf.borrow_mut();
        v.push(rec);
    });
}

/// Reset all thread buffers known to this thread (and our own).
/// Call before a dispatch you want to trace cleanly.
pub fn reset_current_thread() {
    THREAD_TRACE.with(|buf| {
        let mut v = buf.borrow_mut();
        v.clear();
    });
}

/// A copy of every event the calling thread's ring holds, oldest
/// first.
///
/// The ring is left as it is; [`reset_current_thread`] is what empties
/// it. Only the calling thread's ring can be read, because each ring is
/// a thread-local: a worker's events reach stderr through
/// [`request_worker_flush`] and nowhere else.
///
/// A ring records only while [`is_enabled`] answers true and keeps
/// what it recorded after tracing is turned off, so an empty answer is
/// a thread that recorded nothing since its ring was last emptied.
/// Reading a thread's ring for the first time allocates it, as its
/// first event would.
pub fn snapshot_current_thread() -> Vec<TraceRecord> {
    THREAD_TRACE.with(|buf| buf.borrow().clone())
}

/// Per-thread registration. Worker threads call this once at startup
/// so the dump knows who they are. Returns an integer id used in
/// the dumped CSV.
pub fn register_thread() -> u64 {
    THREAD_ID_GEN.fetch_add(1, Ordering::Relaxed)
}

/// Dump the calling thread's trace buffer to stderr as CSV. Other workers
/// have to flush themselves before the dump can include them; the
/// worker loop has a "trace dump" path that writes its buffer on
/// demand.
///
/// CSV columns: thread_name,event_kind_num,payload,tsc
pub fn dump_to_stderr(thread_name: &str) {
    if !is_enabled() {
        return;
    }
    THREAD_TRACE.with(|buf| {
        let v = buf.borrow();
        for rec in v.iter() {
            eprintln!(
                "TRACE,{thread},{ev},{payload},{tsc}",
                thread = thread_name,
                ev = rec.event as u8,
                payload = rec.payload,
                tsc = rec.tsc,
            );
        }
    });
}

/// Worker-loop hook: each worker's loop calls this after it runs a
/// job so the buffer is flushed at a known quiescent point. The
/// `name` is `flynnel-worker-{idx}` typically.
///
/// `examples/trace_dispatch.rs` instead requests a flush with
/// [`request_worker_flush`] and waits on [`worker_flushes_done`].
pub fn flush_with_label(label: &str) {
    dump_to_stderr(label);
    reset_current_thread();
}