Skip to main content

Module bench_guard

Module bench_guard 

Source
Expand description

The quiet-machine guard + pins ledger shared by every perf harness (scoreboard, encoder_scoreboard, enc-kernel-lab).

Factored out of src/bin/scoreboard.rs so the encoder harness cannot drift from the decoder harness on refusal semantics: a benchmark number is only comparable to another benchmark number if both were captured under the same contention policy. The 2026-07-13 GLiNER benchmarking session is the cautionary tale — the same input measured 29.6 ms and 82.9 ms minutes apart on a box at load average 46–209, and two optimization decisions were made on numbers that later turned out to be noise.

Two independent contention axes, because one does not imply the other:

  • CPU — ambient load (compile jobs, browsers) stealing the submitting/worker threads;
  • GPU — a foreign compute process pinning the SMs while uptime reads near-idle (observed on the shared V100 box 2026-07-12: load 3.53 with three foreign 9–18 GiB compute processes resident).

OSFKB_SCOREBOARD_IGNORE_LOAD=1 downgrades refusal to a warning for exploratory runs; the harnesses then refuse --pin, so a contended number can never become the reference every later run is judged against.

Functions§

ensure_quiet_machine
Refuse to measure on a busy machine; returns whether the box was genuinely quiet. A false return is not fatal (exploratory runs under contention are useful) but callers must refuse --pin on it — run in both scoreboards enforces that.
foreign_gpu_procs
Compute processes occupying the GPUs, excluding our own. None when there is no nvidia-smi (Metal exposes no per-process compute query — absence of the tool is NOT evidence of a quiet GPU, so callers must not read None as “clear”).
foreign_load
The load average with our own CPU usage removed — i.e. the contention we did NOT cause.
load_average_1m
1-minute load average, portable via uptime (macOS + Linux). None = unavailable.
nvidia_clock_note
Print NVIDIA clock + throttle state when nvidia-smi is present (informational; pinning absolute numbers on NVIDIA requires locked clocks — nvidia-smi -lgc <mhz>).
parse_compute_apps
Rows of nvidia-smi --query-compute-apps that are not us. Split from the process spawn so the filtering is testable without a GPU.
read_pins
Read a pins ledger; a missing file is an empty ledger (first run on a branch).
self_cpu_seconds
Cumulative CPU seconds this process has burned (user + sys, summed across threads).
write_pins