Expand description
Measure how fast this host’s memory subsystem is right now.
§Why this exists
Benchmark wall times on shared cloud hosts are not comparable across runs.
Measured on this project’s own fleet (2026-08-06, wgs-5M/m7i, one binary,
one input): repeating a replicate on the SAME EC2 instance reproduces within
2.03%, while letting each replicate pick a fresh instance spreads
9.40% — and on one sample the same unchanged binary measured 18.89 s and
25.01 s depending only on which machine it landed on.
That variance is not lost CPU. Across every replicate cpu_time / wall held
at 14.1–14.7 of 16 vCPUs, so the scheduler was giving us our cores; what
changed was that identical work burned ~20% more CPU-seconds. Cores were
stalling on memory, because co-tenants on the same socket were consuming
last-level cache and DRAM bandwidth that the instance contract never
promised us.
So the probe measures memory access rate under whatever contention exists at this moment, and nothing else.
§Why a pointer chase, and why several threads
The probe has to be sensitive to the same resource the workload is sensitive to. bwa-mem’s FMI search is a dependent chain of random accesses over a multi-gigabyte index — latency- and memory-parallelism-bound, not ALU-bound. A tight arithmetic loop in registers would read the same on a contended host as on an idle one and tell you nothing.
A pointer chase reproduces that shape: each step’s address comes from the
previous step’s value, so the CPU cannot prefetch and cannot overlap the
misses within a thread. Running one chain per thread then reproduces the
aggregate memory parallelism of a -t 16 alignment, which is what a noisy
neighbour actually degrades.
§Why it must not resemble the code under test
The probe is deliberately independent of bwa-mem3. If it shared code, then
dividing a measurement by a probe score would cancel real regressions along
with host noise. cpu_time is the cautionary example: it is already a
contention signal, but a genuinely slower binary also burns more of it, so
normalising by it would flatten everything including the thing you are
looking for. A separate pointer chase cannot move when bwa-mem3 changes, and
does move when the host degrades — which is exactly the property that makes
it usable as a control.
§Does it actually respond to a neighbour?
Yes, measured. On an otherwise quiet 12-core host, a 4-thread probe against the same 4-thread probe with an 8-thread memory neighbour beside it:
alone 28.4 / 30.3 / 31.7 M accesses/s 126-141 ns
8-thread neighbour 18.0 / 17.2 / 17.9 M accesses/s 222-233 ns
alone again 29.8 / 29.7 M accesses/s 134 nsA 42% drop, tight within each condition (4.7% spread contended), and it recovers when the neighbour leaves — so the score tracks conditions now rather than drifting. The response is larger than the ~20% wall-time effect it is meant to detect, which is the right way round for a detector.
Two caveats that shape how it should be read. It is not a pure memory probe: 8 spinning CPU-only threads with no memory traffic still cost ~18%, so the score conflates memory contention with CPU availability. For deciding “is this host degraded” that is harmless — either cause means it is — but it matters if the score is ever used as a divisor. And the probe needs a quiet baseline to calibrate against: an early attempt here read 12-22 M/s purely because a build was finishing in the background, which inverted the result until the machine was actually idle.
§Filter first, normalise only once validated
Treating the score as a scale factor assumes probe and workload degrade proportionally. That is plausible and unverified. Treating it as a filter only assumes a bad score means a bad host, which is far weaker. Collect scores next to timings, join on the recorded instance id, and check how much of the between-host variance the score explains before dividing by it.
Structs§
- Probe
Config - How the probe should be run.
- Probe
Result - What the probe measured.