# exatok benchmark method
How exatok measures tokenizer throughput, why the published GPU BPE baselines
are not reusable, and what this harness refuses to report.
Everything here is measured on one machine with one corpus and one vocabulary,
by `src/bench_support.rs` driving `benches/baselines/host_baselines.py`. Numbers
in this document come from that harness. Where a figure is quoted from a paper
it is labelled as such.
## Why this document exists
Two GPU BPE papers report speedups against a tiktoken baseline of roughly
0.94 MB/s:
- BlockBPE (arXiv 2507.11941, ICML ES-FoMo 2025), which also removes regex
pretokenization and accepts a generation-quality loss, so its token ids do not
match a CPU tokenizer's.
- GPUTOK (arXiv 2603.02597), which keeps exact GPT-2 semantics but merges one
pair per pass.
Real tiktoken runs one to two orders of magnitude faster than that baseline.
Measured here, single-threaded and pinned, tiktoken 0.12.0 on `cl100k_base`
sustains roughly 19 to 24 MB/s depending on document size. A speedup computed
against 0.94 MB/s is inflated by a factor of about 20 to 25 before anything
about the GPU is considered.
The failure is not dishonesty. It is that a tokenizer can be measured correctly
and still produce 0.94 MB/s if the timer is placed around the wrong thing, or if
the machine was busy. `benches/baselines/defect_taxonomy.py` reproduces that
number on this machine using real tiktoken on real text, then measures the same
library correctly beside it, so the gap is attributable to methodology rather
than hardware. Run it with:
```
"$HOME/.cache/exatok-bench-venv/bin/python" benches/baselines/defect_taxonomy.py
```
The venv lives OUTSIDE the tree at `$HOME/.cache/exatok-bench-venv`, the
durable path `BENCH_VENV_REL` names in `src/bench_support.rs` and the only
interpreter the sweep's preflight accepts. The scratch corpus lives in the
cargo target directory, which this fleet configures outside the source tree.
Do not put a `target/` beside `Cargo.toml`: repository hygiene deletes an
in-tree `target/` on sight, and a sweep whose corpus is deleted while the
baseline driver is reading it dies by signal with nothing on stderr. Create
the venv once, with the pinned set the Reproducing section below uses:
```
VENV="$HOME/.cache/exatok-bench-venv"
python3 -m venv --system-site-packages "$VENV"
"$VENV/bin/pip" install tiktoken==0.12.0 tokenizers==0.22.2 gigatoken fastokens
```
The consequence for exatok: exactness plus device residency is the defensible
position, and every published baseline has to be re-measured here rather than
cited.
## What is reported
Every row carries all four of these, because a single aggregate MB/s against an
unthrottled state-of-the-art baseline is the specific way this benchmark could
flatter exatok without any individual number being false.
| `MB/s` | Absolute throughput, SI megabytes, bytes of input over wall seconds |
| `GB/s` | The same figure in SI gigabytes |
| `MB/s/core` | `MB/s` divided by `min(threads, physical_cores)` |
| `MB/s/thread` | `MB/s` divided by the requested thread count |
| `par` | CPU-seconds over wall seconds during the timed region |
Both divisors appear so a reader never has to guess which one produced a
figure. On a 16-core, 32-thread machine they differ by 2x, which is enough to
decide whether a reader doubles or halves the number.
`ns/tok` and `B/tok` are also reported. MB/s scales with bytes per token, so
rows at different bytes-per-token are not comparable and the header says so.
### Absolute, never a bare ratio
The table prints no speedup column. GPUTOK's own latency table yields 11.20x
where its text claims 1.67x; a ratio whose inputs are not printed cannot be
checked. Any ratio in a exatok writeup must be computed from two absolute rows
that are both present in the table.
### Active time separate from wall time
`ActiveTime` records `cpu_secs` and an optional `device_secs`, kept separate
from the wall clock. Host CPU baselines have no device timeline, so
`device_secs` is `None` there and is never backfilled from wall time.
`par` (`cpu_secs / wall`) is the derived check that a thread count was actually
delivered rather than merely requested. A 16-thread cell that consumed one
thread's worth of CPU did not run the way its row claims.
### The clock a cell ran at, weighted by occupancy
`mean_mhz` is a whole-machine mean over every logical CPU, so on 32 CPUs a
single-threaded region averages one core at boost with 31 at idle clock. It
tracks how many cores are BUSY far more strongly than the clock the measured
work ran at, and differencing it across cells of different thread count is how
this harness once published a +45.53% "drift" that was occupancy.
`RegionClock` is the probe that claim needs. Per-CPU clock and per-CPU non-idle
time are sampled at both ends of each cell's own timed region, and the two clock
readings are averaged over only the CPUs that did the work, weighted by how long
each was busy, using the SAME weights at both ends. A difference between the
endpoints is then attributable to boost state rather than to which cores were
occupied. Measured on this fleet at one instant: 4568 MHz busy-weighted against
3274 MHz whole-machine mean, a gap of 1294 MHz.
Both figures are reported per cell. `region_clock` is `null` when the per-CPU
probes are unavailable, which is a different statement from a clock that held
steady.
## Machine
Recorded per run rather than assumed, by `Machine::probe`.
| CPU | AMD Ryzen 9 9950X, 16 physical cores, 32 logical |
| Governor | `performance` |
| Memory | 96 GiB |
| GPU | NVIDIA RTX 5090, 32 GiB, driver 570.211.01, compute capability 12.0 |
The report also records mean clock at the start and end of the sweep and flags
drift at or above 5%, so a run that lost boost clock partway cannot silently
shift late measurements relative to early ones.
## Corpus
One corpus file, built by a deterministic sorted walk of the Santh tree, hashed
with `corpus_digest` (two FNV-1a passes under different seeds, reusing
`crate::fnv1a32`). Every library in a run reads byte-identical input, and the
digest is printed so two runs can be compared or shown to differ.
Real prose and code, not repeated synthetic text: naively repeated text
compresses far better under BPE and inflates MB/s for every library at once.
Measured bytes per token on this corpus at `cl100k_base` is about 3.69 to 3.86.
## Regimes
- **single**: one document, one thread. The latency-bound case. A single
document cannot be split across threads without changing the answer, so this
is the honest per-core number.
- **batch**: many documents filling a byte budget, encoded at 1, 16 and 32
threads. The throughput case.
Document sizes sweep from under a kilobyte to about 15 MB, so a per-call
overhead that only shows up on small inputs is visible as a curve rather than
hidden in one aggregate.
### Thread pinning
`taskset -c` pins runs, and the pinning set is recorded per run:
- 1 thread: CPU 0.
- 16 threads: one logical CPU per physical core, so no two threads share a
physical core.
- 32 threads: unconfined across all logical CPUs, to measure SMT scaling.
`RAYON_NUM_THREADS` and `TOKENIZERS_PARALLELISM` are set before any tokenizer
import, because rayon reads its thread count once when it builds its global
pool. Without this, `tokenizers` and `gigatoken` size their pool from the
machine's logical CPU count regardless of what a cell requested: under
`taskset -c 0` that put 32 rayon workers on one permitted CPU, so a cell
labelled 1 thread was really 32 threads time-slicing one core.
### Timed region length
Work is repeated inside the timed region until it clears 0.5 s, and the
repetition count (`xreps`) is reported. The count is calibrated against a
**warm** measurement, not a cold first call: the cold call pays first-touch page
faults and cache fill, so it overestimates steady-state cost and therefore
underestimates the repetitions needed. Calibrating on it produced 0.188 s
regions against a 0.5 s target.
`input_bytes` records the size of one unit and `xreps` the repetitions, kept
separate so the reported input size is reproducible across machines instead of
absorbing a machine-dependent loop count.
## What the harness refuses to report
`assess_plausibility` classifies every cell. Five verdicts fail the run, four of
them because the harness measured the wrong thing rather than because a library
was slow; three are results.
| `ShortTimedRegion` | yes | Timed region under 0.25 s, so its throughput means nothing |
| `ThreadsNotHonoured` | yes | Achieved under half the requested threads |
| `Contended` | yes | A pinned 1-thread cell got under 0.9x of one core |
| `ForeignLoad` | yes | Over 5% of one core was consumed elsewhere on the machine |
| `SuspectedMeasurementDefect` | yes | Under the per-core floor with no recorded explanation |
| `KnownSlowLibrary` | no | Under the floor for a reason established by experiment |
| `PerCallBound` | no | Input too small for dispatch overhead to amortize |
| `Plausible` | no | Clears every gate |
### The floor is per core, not per thread
`PLAUSIBLE_FLOOR_MB_S_PER_CORE` is 5.0 MB/s and divides by
`min(threads, physical_cores)`.
An earlier version divided by the thread count and flagged 25 cells on the first
full sweep, most of which were correct. Putting 32 threads on 16 physical cores
must roughly halve per-thread throughput; that is arithmetic, not a defect. A
guard that reports correct SMT scaling as a measurement error trains its reader
to ignore it.
Falling under the floor is a question, not a verdict. A library can legitimately
be that slow, and `KNOWN_SLOW_BASELINES` records which ones are, with the
evidence. An entry there suppresses nothing: the cell still reports its measured
number with the reason printed beside it. Lowering the floor instead would also
hide a genuine regression in a library that is not slow.
### Contention is a measurement error, not a slow tokenizer
This is the check that protects the comparison's honesty, and it is the one
whose absence produces numbers like 0.94 MB/s on a correctly written benchmark.
Contention depresses the **baseline**. A depressed baseline inflates every
exatok-versus-baseline ratio computed against it, and it does not look like a
failure: it looks like a flattering result, so nothing prompts anyone to
question it.
A single-threaded cell pinned to one CPU consumes one CPU-second per wall second
unless the kernel descheduled it for something else. Below 0.9 the cell is
flagged `CORE NOT EXCLUSIVE` and the run refuses to publish.
The baselines in this document were measured while other work was running on the
same box. The exclusivity claim is retracted and every host figure here is a
lower bound. See "An advisory is not an interlock" below.
#### The exclusivity check is two gates, not one
`cpu_secs / wall >= 0.9` detects descheduling. It does **not** detect two other
ways a baseline can be starved:
- An SMT sibling running on the partner of the pinned core.
- Memory bandwidth contention from elsewhere on the machine.
Both leave the process on its assigned CPU at full CPU-seconds while its
throughput drops, so that ratio stays near 1.0 and says nothing.
The driver therefore measures foreign CPU directly: it samples `/proc/stat`
around each timed region, subtracts this process's own CPU time, and emits the
remainder as `foreign_cpu_secs` per cell. `assess_plausibility` divides it by
wall seconds and voids any cell over `MAX_FOREIGN_LOAD_RATIO`, 5% of one core,
as `ForeignLoad`. That covers both modes above, because a competing process
still has to consume CPU somewhere on the machine even when it is not stealing
this core.
`MAX_FOREIGN_LOAD_RATIO` in `src/bench_support.rs` and the constant of the same
name in the driver must stay equal. The driver names the contended cell in prose
and the host decides its fate, so a divergence either prints a note nobody acts
on or refuses a cell with no note explaining why.
The foreign check runs **before** the thread gates, because foreign load also
depresses achieved parallelism. Checking threads first reports a starved cell as
a thread-delivery bug, which is what happened to eight 16-thread cells: see "An
advisory is not an interlock".
## Baselines
Every baseline reads the same corpus and the same vocabulary, and its ids are
checked against a reference on a 64 KiB sample. A library whose ids disagree is
still reported, with its row marked, because a throughput comparison across
libraries that emit different ids is not a comparison.
| tiktoken | 0.12.0 | Reference for id agreement |
| gigatoken | 0.10.0 | CPU state of the art; same `.tiktoken` rank file as tiktoken |
| hf-tokenizers | 0.22.2 | Timed through `encode_batch_fast` |
| fastokens | 0.2.0 | Unavailable, see below |
Vocabulary rank files are materialized durably in `~/.cache/exatok-vocab/`
(`cl100k_base`, `o200k_base`, `p50k_base`, `r50k_base`) so a sweep does not
depend on the network.
### hf-tokenizers is timed on its ids-only path
`encode` and `encode_batch` build character offsets, an attention mask, type ids
and a word-id map for every token. tiktoken's `encode_ordinary` returns a list
of ints. Timing one against the other compares output formats, not tokenizers.
Measured here, the offset path is 1.41x slower at 60 KB and 1.99x slower at
1 MB.
`encode_batch_fast` returns ids only and is what the harness times. The
exactness check still goes through `encode`, where ids are what matter and speed
is irrelevant.
Even on the fast path, hf-tokenizers measures roughly 8.1 MB/s per core at 60 KB
falling to 4.6 MB/s per core at 1 MB, which is below the floor. That is recorded
in `KNOWN_SLOW_BASELINES` with those numbers rather than suppressed, and it is a
real property of the library on this vocabulary: the throughput degrades with
document length on both APIs.
### fastokens is recorded unavailable, not estimated
`fastokens` 0.2.0 raises `ValueError: model error: unsupported model type:
Other` on the `tokenizer.json` this sweep uses. It is therefore reported as
unavailable with that exact message. It is not estimated, not omitted, and not
substituted with a different vocabulary, because a row measured on a different
vocabulary is not comparable to the others.
## Reproducing
```
# The baseline libraries live in a DURABLE venv outside the tree. NOT under the
# cargo target directory: a venv kept there was deleted between two sweeps,
# which left a published gigatoken figure unreproducible. tiktoken and
# tokenizers are pinned, because exactness and the published baselines are
# stated against those versions; gigatoken and fastokens carry no pin, because
# the harness checks only that each module imports.
VENV="$HOME/.cache/exatok-bench-venv"
python3 -m venv --system-site-packages "$VENV"
"$VENV/bin/pip" install tiktoken==0.12.0 tokenizers==0.22.2 gigatoken fastokens
# Full host baseline sweep. Needs a quiet machine; minutes long.
CARGO_BUILD_JOBS=1 cargo test -p exatok --test bench_support -- \
--ignored --nocapture host_baseline_sweep
# The harness's own tests, including the plausibility gates.
CARGO_BUILD_JOBS=1 cargo test -p exatok --lib bench_support
CARGO_BUILD_JOBS=1 cargo test -p exatok --test bench_support
# Reproduce the published sub-1 MB/s figure, then measure it correctly.
"$VENV/bin/python" benches/baselines/defect_taxonomy.py
```
Every sweep writes a UTC-timestamped report to
`$HOME/.cache/exatok-bench/reports/host-baseline-<stamp>.json`, whether it
passed or failed, carrying every row, the machine description, the corpus digest,
and per cell the foreign load, achieved parallelism, best-case seconds, raw
per-repetition samples and region clock. `--json PATH` writes an additional copy
wherever asked.
A failing sweep is the one whose per-cell numbers someone will need. An earlier
version wrote a report only when passed `--json`, so the first question asked of
a refused table, which cells failed and why, could be answered only by rerunning
the whole sweep.
If the venv is missing the sweep REFUSES, naming the expected path and the
command above, rather than recording one unavailable row per library. Six silent
unavailables read like six libraries that do not exist on this platform. They
were one deleted directory.
## Failures this harness caught
The primary result of building this harness is not the table. It is that three
full sweeps were stopped before publishing anything, two by a gate and one by a
kill the harness at first could not name, and that every defect found was the
same class of error that produces a published 0.94 MB/s tiktoken baseline.
Recorded here because a guard that fired and was believed is worth more than a
clean run, and because a reader who does not know these were caught will
reintroduce them.
### Sweep 1, rejected: two driver defects and one guard defect
`MIN_TIMED_SECS` was 0.02 s, so a 33.5 MB batch completed in about 19 ms and
gigatoken recorded 4.04x parallelism on 16 pinned cores. At that duration the
timed region is substantially rayon pool startup, charged to the tokenizer.
Raised to 0.5 s, with a steady-state calibration loop so a warm pass cannot
under-run it.
hf-tokenizers was timed through `encode`, which builds character offsets, an
attention mask, type ids and a word-id map per token, against tiktoken's
ids-only `encode_ordinary`. That compares output formats, not tokenizers. Now
timed through `encode_batch_fast`.
The third defect was in the guard, not the driver, and is the more interesting
one: the floor divided throughput by requested thread count, so it flagged 25
cells of which most were correct SMT behaviour. 32 threads on 16 physical cores
*must* halve per-thread throughput. A guard whose false positives are the
expected physics of the machine gets relaxed wholesale, which is worse than
having no guard. Hence `min(threads, physical_cores)`.
### Sweep 2, rejected: serial marshalling, and the freeze biasing the sweep
Eight cells failed `ThreadsNotHonoured`, every gigatoken thr=16 cell plus two
tiktoken ones, achieving 1.28x to 4.78x against 16 requested. Diagnosis: with
up to 35508 documents per batch the timed region is dominated by serial
Python-level result marshalling (`sum(len(x) for x in ...)` over that many
returned lists) rather than by tokenization. Token counts are deterministic and
must be computed once outside the timed region, leaving the timed call free to
discard its result.
The second finding is a defect in this harness's own instrument, and it
retracts a claim I published from the same run. The drift detector fired at
+45.53%, 3734 MHz "mean clock at sweep start" against 5435 MHz at the end, and I
reported that as thermal ramp: an idle machine sits at low clock, so an
exclusive-access freeze would guarantee the first cells are measured cold and
the last at full boost. That mechanism is wrong, and stating the predicate
beside the number is what refutes it.
`mean_mhz()` averages `cpu MHz` over EVERY logical CPU in `/proc/cpuinfo`. On a
32-thread machine a single-threaded cell has one core at full boost and 31 at
idle clock, so the mean is dominated by HOW MANY CORES ARE BUSY, not by the
clock the measured work actually ran at. Comparing the sweep's first record
against its last therefore compares cells with different thread counts and
different regimes. It measures occupancy and reports it as drift.
Measured, after moving the probe from record-emit time into the timed region so
it at least samples while the cores are hot: per-cell readings across one
three-size sweep span 1925 to 5125 MHz, a wider spread than the +45.53% that
was called a finding, and they track thread count rather than position in the
run. The original probe was worse still, sampled after the timed region during
the Python-level gap between cells, where the machine has already dropped to
idle.
So the honest status: this harness cannot currently substantiate a claim about
boost-clock drift, and any cross-library ranking must instead rest on the
per-cell gates that do not depend on this probe (`par`, foreign CPU-seconds,
timed-region length). A usable clock probe has to average only the CPUs that
were busy during the region, or compare exclusively between cells of equal
thread count. Neither is implemented.
What survives, and is the reason the ramp below is still worth having: a sweep
starting from genuine idle does begin cold. The ramp drives all-thread load
until successive readings settle, measured at 2890 rising to 4574 MHz before
the first timed cell on one run, and at +1.41% on an already-warm machine. That
removes a real cold start. It does not license the drift claim, and the ramp is
not evidence for it.
The general shape, which is not specific to tokenizers: the precondition
requested to protect a measurement (a quiet machine) is also a confound,
because isolation and steady state are different properties. And the sharper
one, learned the hard way here: I diagnosed a mechanism from a single
start-to-end delta without asking what the probe averaged over. A number can be
real, reproducible, and about something other than what its name says.
### Sweep 3, killed: an external reaper, reported as a silent clean failure
Three consecutive sweeps died after several minutes with exit status `137` and
an empty stderr. The harness reported the failure as `exited with None`, because
`ExitStatus::code()` is `None` for a signalled process. `None` beside an empty
stderr reads like the driver failed silently and cleanly, so the investigation
went looking for a bug in the driver: a memory leak on the largest cell, then an
out-of-memory kill. Both were wrong. `dmesg` held zero OOM lines, `systemd-oomd`
had no entries, and memory pressure sat at 0.30%.
The cause was a cron CPU watchdog on this machine that sends `kill -9` to any
non-allowlisted process holding more than 95% CPU for five minutes, and logs the
kill. Its log named all three victims at exactly `360s high CPU`. A correct
single-threaded benchmark over a 64 MiB corpus pins one core for longer than
that by construction, so the measurement and the reaper were in direct conflict
and the reaper always won.
Two fixes. The diagnostic now names the signal and points at the reaper logs,
because a signal number is the whole diagnosis and `None` is not. The driver is
allowlisted by its exact script path, matched against the full argument vector
rather than the executable name: the existing allowlist tested only `ps aux`
field `$11`, so a `python3` resolved through `PATH` reported just `python3` and
could never carry a project token, and matching loosely on arguments instead
would have exempted anything with `cargo` in its argv, since the corpus path
contains `cargo-target`.
The scratch corpus moved out of the source tree in the same change, for a
related reason. It had been written to `<crate>/target/bench-corpus.txt`, an
in-tree `target/` that repository hygiene deletes on sight, and it was in fact
deleted from under a running sweep. A 64 MiB input that a routine cleanup may
remove mid-measurement is not an input.
The general shape: a benchmark's real preconditions include the machine's own
policy, not just its hardware and its quiet. And a failure that names nothing
will be attributed to whatever the reader was already suspicious of. The signal
number cost minutes to surface and would have cost hours to guess.
### Prevented, not rejected: a suite run launching the sweep
`cargo test --all-targets` runs the bench binary. `benches/host_baseline.rs`
gated its self-check on a `--test` flag, on the belief that cargo passes one to a
`harness = false` bench. It does not. Read from `/proc/<pid>/cmdline` while the
sweep was live: `argc == 1`, argv holding the binary path and nothing else, with
the driver's own 20-argument command line in the same read as a control proving
the instrument prints arguments when they exist. The gate was unreachable for the
file's whole life, so a package-wide suite run spawned the full 4M-token sweep
and held the machine for 11 minutes.
That is a measurement defect and not merely a slow test. Every measurement taken
anywhere on the machine during those 11 minutes was contended, which is the
condition `Contended` exists to reject, and the sweep's own numbers would have
been taken while every other target in the package competed for the same cores.
The sweep is therefore opt-in behind `--sweep`, never opt-out. Both automated
verbs were then measured rather than assumed:
| `cargo test --all-targets`, `--benches` | `[]` | self-check, 5.9 s |
| `cargo bench --bench host_baseline` | `["--bench"]` | self-check, 4.0 s |
| `cargo bench --bench host_baseline -- --sweep` | `["--sweep"]` | the sweep |
The measured `["--bench"]` is why the flag cargo already passes cannot serve as
the opt-in: gating the sweep on `--bench` would have left `cargo bench` starting
it automatically, trading one unmeasured argv assumption for another.
### An advisory is not an interlock
The claim "measured on an otherwise idle machine" was carried by an `#[ignore]`
attribute string asking a human for exclusivity, and by a sentence in this
document asserting a machine freeze. Neither is enforcement. The sweep that
produced the host figures ran for 780 s while other processes worked on the same
box, so that sentence was false when written and the figures are lower bounds.
Two defects made it publishable, and only the first was known:
1. The gate cannot see foreign load. `cpu_secs / wall` asks whether this process
kept its pinned CPU, not whether the machine was busy. Recorded above as a
known limitation before the sweep ran.
2. The mitigation for that limitation reached no reader. The per-cell
`foreign_cpu_secs` was computed, emitted, and then dropped by the parser,
while its `CONTENTION` note went to the channel `note`'s own docstring calls
"ignored by the record parser". The evidence existed per cell and landed in
no artifact anyone reads, so the gate could not refuse a contended
measurement. It could only fail to notice one.
A cell starved by an SMT sibling or by memory bandwidth was therefore published
as a slow tokenizer. That is the wrong class: not a slow run, a non-run, and a
harness that cannot tell those apart reports the second as the first.
Both are now closed. `foreign_cpu_secs` is a required record field and
`ForeignLoad` voids the cell rather than annotating it, ahead of the thread
gates. The figures already published stay lower bounds: a gate cannot
retroactively clean a machine, and replacing them needs a genuinely idle box.
The first diagnosis of the eight refused 16-thread cells was also wrong, and the
data refuting it was published alongside it. The cells were attributed to a
`taskset` and `RAYON_NUM_THREADS` interaction. A CPU mask and a rayon pool are
fixed for the life of a process, so a misconfiguration produces a constant
achieved parallelism across every cell of one invocation. The measured spread
within one invocation was 1.24x to 4.71x for gigatoken and 3.95x to 7.45x for
tiktoken. A constant cannot produce a 3.8x spread; load varying across a 780 s
window can. The refusal was correct and its stated cause was not.
Stated in the direction that costs this harness rather than the direction that
flatters it: contention only ever depresses a baseline, so the tiktoken figures
here understate tiktoken and the comparison against the published 0.94 MB/s
survives unharmed. The same error pointed the other way, a contended exatok
cell, would flatter exatok. That asymmetry is why the fix is a gate rather than
a footnote.
### Sweep 4, refused: 56 of 56 cells on a contended box
The strongest evidence for the foreign-load gate came from a run where it
refused everything. Fifty-six measured cells, and the gate voided all fifty-six.
Foreign CPU load ran from 1.607 to 14.163 cores' worth of work this process did
not create, median 8.062. All 56 exceeded the 0.05 threshold, and 39 still
exceeded a hundredfold looser one, so the threshold was not the marginal call it
would first appear to be. Both alternative explanations were ruled out from the
same report: best-case time was 0.4282 s against the 0.25 s floor, so ZERO cells
were short-timed, and sweep clock drift was 0.011, flagged not significant.
Correctness was untouched throughout, every measured library still producing ids
identical to the reference. Contention corrupted the timing and nothing else.
The detail worth keeping is the one a per-thread efficiency check cannot reach.
Minimum achieved parallelism across the single-threaded cells was 0.9938, and
NONE of them failed the 0.9 single-thread gate. One cell recorded achieved
parallelism 1.0004 against foreign load 8.8461. A benchmark can keep its whole
core and still be measuring a machine doing eight cores of other work, so
`cpu_secs / wall` reports a clean run while throughput is depressed. That pair,
1.0004 against 8.8461, is the clearest single artifact of why the foreign-load
gate exists and why it is checked before the thread-delivery gates.
Quantified, because the size of the error matters: tiktoken single-thread read
13.05 to 25.07 MB/s in this run against 17.62 to 25.04 MB/s measured earlier, a
26% understatement at the low end. Every host figure taken under these
conditions is a LOWER BOUND.
Thirteen of the 56 cells also exceeded 20% relative standard deviation, and 36
exceeded the 5% advisory, with a maximum of 0.4140. Those 13 would void on
variance alone, independent of foreign load, and best-of-N over them is
meaningless rather than merely noisy. They are recorded here because they are
INVISIBLE in the table: `ForeignLoad` is checked first and returns early, so no
cell in this run ever reached the variance gate. On a box quiet enough to clear
0.05 foreign load but still bursty, those are the cells that would otherwise
publish a best-of-N that means nothing.
Two libraries were unavailable in this run rather than measured. gigatoken and
fastokens both raised `ModuleNotFoundError` because the venv had been deleted,
the failure the durable path and the loud preflight now prevent. The consequence
is that the gigatoken figure of 4.115 GB/s on 16 physical cores, published from
an earlier run, is not re-derived here and is to be treated as UNVERIFIED until
it is measured again on a quiet machine.
Disclosed rather than left to be discovered: `EstimatorDependent` was added to
`is_harness_defect()` about an hour before this refusal was reported, so a new
gate belonging to this harness was itself a live candidate for the failure. It
was checked first. Every one of the 56 defects was `ForeignLoad`, none were the
new gate, and zero cells carried a device timeline so the device gate could not
have fired either. Self-implication before external explanation is the order
that keeps the rest of a diagnosis worth reading.
## Reporting rules for exatok's own numbers
1. Absolute MB/s and GB/s, with per-core beside it. Never a bare ratio.
2. Active device execution time separate from wall time. A GPU figure that
includes host-side transfer and allocation is a different measurement from
one that does not, and both are worth reporting, labelled.
3. A configuration that is not shippable is labelled measured-but-not-shipped
and never listed beside shipping numbers. Multi-merge is the case in point:
its exactness rests on a training-time property of frequency-ordered ids that
no loader can check, so it is permanently off, and a table that lists it as a
exatok result is how a future reader switches it on.
4. Any cell that fails a harness gate is not published. It is re-run.
5. A quiet machine is a PRECONDITION for a certified host table, not a nicety.
Foreign load is measured and recorded per cell, and a contended sweep is
published as a refusal plus its measured foreign load, never as a softened
bound. Where a box cannot be quieted, an absolute figure is not certifiable,
and only a paired before-and-after with interleaved ordering is defensible.
6. A GPU figure taken while another process holds the card is not certified, and
exclusive device access is a PRECONDITION for a certified absolute device
figure rather than a nicety. Two mechanisms enforce it, because a booking
convention between agents is not a mechanism. First, `DeviceLock` takes an
exclusive `O_EXCL` lock at `$HOME/.cache/exatok-bench/device.lock` around the
timed region only, with staleness detection keyed on the holder's pid so a
process killed before its `Drop` runs cannot wedge the card. Second,
`sample_device_load()` reads per-PID residency and SM utilization at both
region boundaries, and a round whose foreign residency climbs more than
`DEVICE_RESIDENCY_TOLERANCE_MIB` above baseline is voided and retried. Note
what the lock does NOT cover: it is advisory between harness callers, so a
plain `cargo test` on the same tree ignores it entirely. The residency probe
is what catches that case.
7. The device probe is `nvidia-smi pmon -s um`, not `--query-compute-apps` (which
showed an 87.5% false-absence rate during live CUDA runs and was retired as an
all-clear observable) and not `--query-gpu memory-used` (which reports a
whole-device total with no per-process attribution, so on a box running an
external inference server it reads tens of GB and says nothing about whether a
given region was contaminated). Foreign SM percent is sampled at roughly 1 Hz,
so a blank column means "no kernel in this window" and not a measured zero: it
is reported as evidence and never gated on.
8. The device-contention gate applies only to cells that used the device.
Scoping is automatic, from this process's own device residency, rather than
from a caller-supplied flag: a flag anyone can forget is not a safety
mechanism. Unscoped, the gate refused a CPU-only region because an external
inference server's residency drifted from 24764 to 25418 MiB, load that shares
neither SMs nor device bandwidth with host work. A gate that blocks work it
does not apply to trains people to disable gates.
9. The estimator depends on the statistic. An ABSOLUTE cell quotes the MINIMUM of
its samples, because contention is strictly additive and the fastest
observation is the least contaminated. A PAIRED ratio quotes the MEDIAN of the
per-round ratios, because the contaminant has largely cancelled and taking the
best ratio selects the luckiest pairing rather than estimating the central one.
Applying best-of-N to a multi-cell decomposition manufactures wins: with eight
phases it is eight independent chances at a favourable outlier. Paired
comparisons need at least `MIN_PAIRED_ROUNDS` (8) interleaved rounds and a
delta spread within `MAX_PUBLISHABLE_REL_STDDEV` (0.20); a wide spread is an
uncontrolled measurement, not a small effect.
10. Every figure carries its PROVENANCE: `probed` (observed on this machine, this
run), `derived` (computed from probed values), or `fixture` (a hand-written
constant, a synthetic profile, or a reference figure from someone else's
paper). Only `probed` may be quoted as a measurement. This exists because five
numbers in this project read as measured and were not: a test pinning 174,080,
a doc comment asserting occupancy intent for a heuristic with no occupancy
term, a synthetic device profile named after this GPU overstating its threads
per SM by a third (2048 against a measured 1536), an unqualified test count,
and a per-crate API gate reporting byte-stable for a crate it does not inspect.
Every one was correct when taken and became false when its origin was lost, so
the origin is persisted next to the value rather than remembered.
11. A test count without its feature set is not a measurement. `cargo test -p
exatok --lib` reports 255 on default features and 286 with `confluence`,
because 31 unit tests live behind that flag; both are correct and they are not
comparable. Comparing two such counts read a benign packaging change as a
17-test regression. `TestCount` cannot render the bare number, and
`comparable_to` refuses a cross-feature comparison. The feature set is probed
from `cfg!` at report time rather than stored, so it cannot disagree with the
binary that produced the numbers.
12. A benchmark's setup belongs outside its timed region, and a cell that
measures a different operation than its name says is worse than a slow one.
Both halves were live defects here. The device throughput cells uploaded the
entire 16 MiB resident merge table inside `bencher.iter`, costing 2.17 to
2.75 ms per iteration by median (1.96 to 5.54 across min and max) in the cells
this design's throughput claim rests on; they now bind a session once, outside
the region, which is what production does. The launch-overhead floor cells were
worst affected, since their target has a fixed base near 12 us and the upload
was on the order of 180x that, so they could not resolve it at all. They were
NOT fixed by rewriting `encode` as `encode_batch(&[input])`, which would have
harvested the same saving while changing the measured operation under an
unchanged cell name: `encode_batch` flattens pretokens across input boundaries
and keeps per-input bookkeeping. Any device figure predating that hoist is not
comparable with one after it.
13. A ratio names its faster arm in words. Printing the orientation is not
enough: the first paired device capture printed `2.2260x` beside `0.014983 s
(host), 0.006621 s (device)` under the caption "above 1.0 means the HOST is
faster", which reversed all eight cells while every number in them stayed
correct. That is the worst failure mode available to a report, because nothing
looks wrong. `Paired::render` therefore computes the winner from the ratio and
prints `faster arm : device by 2.2260x`, reciprocating so the losing direction
reads as `host by 4.0000x` rather than `0.25x`. The direction is stated once,
by the type that owns the ratio, and call sites are forbidden from restating
it: two captions in two places are two chances to disagree.
14. A ratio whose round range straddles 1.0 has an UNRESOLVED SIGN and is
reported as unresolved, not as a small win. The cjk cell read a median of
1.0138x across a 0.9330 to 1.1280 range, so rounds exist in which each arm
won and the 1.4 percent is noise being quoted as a result. Publishability and
sign are separate questions: that cell passed the spread gate (rel stddev
0.0604) and still could not say which arm was faster. A repeat capture
settled the matter in the direction that vindicates the rule rather than the
median: cjk came back at 0.8125x, a HOST win, so the earlier median had the
sign backwards and only its range disclosed the risk. A median on the
favourable side of 1.0 is a direction only when the whole range agrees with it.
15. Kernel time is not a predictor of end-to-end throughput, and a kernel figure
is never published as though it were one. Measured across four shapes at
262144 B, the shape with the FASTEST kernel was the only shape the device
clearly lost: `short_pretokens` ran 797 to 922 MB/s in-kernel at about 3
percent of wall and lost to the host by 3.05 to 3.12x, while `prose` ran a
mediocre 295 to 317 MB/s at about 10 percent of wall and won by 1.98 to
2.23x. The device advantage tracked how slow the HOST path was on a shape,
not how fast the kernel was. So a kernel MB/s is reported with its
share of wall beside it, always per shape and never aggregated, because the
same kernel number supports opposite conclusions at 3 percent and at 90
percent of wall.
16. A SIGN needs two independent captures, because the spread gate measures
precision within one run and not reproducibility across runs. Two full
captures of the same four shapes, each passing every gate, disagreed on which
arm wins cjk. The host arm reproduced tightly (-2.4 to +5.1 percent per
shape), while the device arm came back 11.1, 11.5 and 12.0 percent slower on
three shapes and 31.0 percent slower on cjk, which is what flipped it. Note
the scale mismatch that makes this a rule rather than an anecdote: cjk's
within-run rel stddev was 0.0397, and its between-run device drift was 0.31,
roughly eight times larger. A gate reading only the first number certifies a
figure it has no information about. So a directional claim is published only
where two captures agree, and an absolute is published with its observed
between-capture drift beside it.
THE CAUSE, corrected, because the first diagnosis here was wrong and the
correction is the instructive part. A one-directional shift on every shape was
attributed to DEVICE state (clocks, thermals, foreign residency). The evidence
says HOST state: the capture ran while the agent swarm rebuilt the tree, and
its own log opens with `Blocking waiting for file lock on artifact directory`.
The mechanism is why only one arm moved. The device arm is 89 to 97 percent
host orchestration over thousands of small latency-bound driver calls, so it is
preempted by competing rustc; the host arm is a throughput-bound loop that
needs one core and gets it. Hence a device-arm-only slowdown with a sign, and
"noise does not have a sign" correctly detected a state change while naming the
wrong state.
That the harness could not tell these apart was the defect. It gated foreign
DEVICE load and recorded NO host load, while `time_region` had been measuring
`foreign_cpu_secs` all along and the paired path discarded it. The data was
already in the process and the instrument threw it away, then reported
confidently without it. Host load is now a recorded field on every paired
round, and the gated quantity is its STATIONARITY
(`MAX_PAIRED_HOST_LOAD_DRIFT`, 0.25) rather than its level: a paired ratio
cancels a contaminant present in both arms of every round and cannot cancel one
that arrives or leaves mid-comparison. Gating on level would refuse a valid
capture taken on a uniformly busy box and would still have admitted the capture
that caused this, whose load CHANGED partway through. An unrecorded reading
renders as NOT RECORDED and never as an idle machine.
17. Every published duration carries the WORK COUNT that produced it, and a cell
whose work is absent or unstable is refused rather than reported. A duration
alone cannot separate a real speedup from an arm that quietly stopped doing
the work, because both read as fast, and the faster arm is then the broken
one. The same hazard was measured from the other side elsewhere in this
effort: a gate fell from 1064.93 s to 3.39 s as the machine drained, and what
made the fast run trustworthy rather than suspect was a pinned
`total == expected_total`, which makes a vacuous pass arithmetically
impossible instead of merely unlikely.
So `compare_cells` records one unit's bytes and tokens for EACH arm, prints
them beside the durations, and refuses two cases it used to accept: an arm
that consumed zero bytes, whose duration measures nothing at all, and work
whose count changes between rounds, where a single printed ratio divides one
quantity by a different one while looking entirely ordinary. The second is
the one that matters over a long capture, since drift mid-run is invisible in
the output it produces.
The pin compares each arm against ITSELF across rounds, never across arms, so
two arms may legitimately produce different token counts and cross-arm
agreement remains the job of the exactness table rather than of this gate.
A count that nothing checks is decoration, which is why the refusals exist
and not merely the printed figure.