# bench
Two harnesses, measuring different things. Neither is a substitute for the other, and
knowing which one answers your question is most of the value here.
| `bench/throughput.sh` | end-to-end req/s and req/CPU-second against nginx | when the machine differs from the one a quoted number came from |
| `cargo bench` (criterion) | per-call cost of `handle_request` and path resolution | when the change you made is *outside* those two functions |
## Throughput, against nginx
```bash
./bench/throughput.sh # both servers
SERVER_ONLY=1 ./bench/throughput.sh # skip nginx
DURATION=30s CONNECTIONS=100 ./bench/throughput.sh
```
Needs `oha` (`cargo install oha`) and, for the comparison, `nginx` on `PATH`. It prints
the host, core count, nginx version and crate version alongside the result, so any figure
copied out of it can be checked against the machine that produced it.
**The metric is req/CPU-second, not req/s.** Throughput can be bought with cores;
efficiency cannot. Both are printed, and the CPU column should read ~100% for a
single-worker server under saturating load — if it does not, the run was not saturating
and the numbers mean nothing.
### Fairness rules, all load-bearing
- **One worker each.** The server example uses `#[tokio::main(flavor = "current_thread")]`;
nginx is configured `worker_processes 1`. A comparison between a multi-threaded server
and a single-worker nginx measures the machine, not the software.
- **Identical files**, served from the same directory, generated by the script so the two
cannot drift apart.
- **A plain server.** `examples/bench_server.rs` enables nothing — no SPA mode, no live
reload, no immutable-asset predicate. Each of those adds per-request work, and a
baseline should measure the floor rather than one deployment's feature set.
- **Keep-alive on both sides**, which is the default for HTTP/1.1 and for `oha`. A server
that closed per request would look 4x slower for a reason unrelated to its file serving.
- **CPU is read from the server process's own accounting** (`ps -o cputime`), sampled
either side of the measured window, so the load generator's CPU is not counted against
the server. For nginx that is the *worker* pid; the master only supervises.
### Three ways this harness was wrong before it was right
Recorded because each produced a confident, plausible, wrong number:
1. **No warm-up: the first measured run read 14,704 req/s where every subsequent run read
60,475.** Cold binary pages and a cold page cache. Recording that first number would
have reported a catastrophic regression that did not exist. There is now a discarded
3-second warm-up before the measured window.
2. **`oha` reports `Total: 3000.7195 ms`, and the script read it as seconds.** A clean
1000x error, which made CPU% round to zero and req/CPU-second read as 74 *million*.
The unit is now parsed.
3. **The original 56,700-vs-52,500 figures named no machine.** They were quoted in
`README.md` as a flat competitive claim for months. Re-run on a Mac16,2 the same crate
serves 60,877 req/s while nginx reaches 74,566 — the claim inverts, not because the
crate slowed but because nginx scales better on that hardware. An unreproducible
benchmark is not evidence, and this is what `mini-serve`'s bench README means by
"record the date and hardware with any figure quoted elsewhere, since both move."
## Where the gap to nginx actually is
Measured on Mac16,2, one worker each, a 484-byte file, `oha` c=50. Per-request CPU
microseconds, derived from req/CPU-second at a saturated core:
| the same bytes from memory, no filesystem at all | 196,360 | **5.09** | the HTTP stack, and nothing else |
| a real file: open + fstat + read | 66,077 | 15.13 | **+10.04** filesystem |
| the same, plus `.br`/`.gz` sidecar probing | 58,638 | 17.05 | **+1.92** two failed `open()`s |
| nginx (66k–76k across runs, see below) | ~70,000 | ~14.3 | — |
**The HTTP stack is not the bottleneck.** Serving from memory reaches 196k req/s — roughly
2.5x nginx's entire request path. About 70% of a real request's cost is filesystem
syscalls, and `mini-serve` plus hyper account for the other 30%.
### The three levers, in measured order
1. **Avoid the sidecar probe when it cannot succeed: +11–13%** — **done**, as
`Server::without_precompressed()`. Every request opened `index.html.br` and
`index.html.gz`, both absent, because browsers send `Accept-Encoding` on every request.
Measured two ways: 58,638 → 66,077 req/s with `accept-encoding: identity`, and a ~11%
mean improvement interleaved with the flag.
Left **off by default is what it is not**: the default still probes. Inferring the answer
from a directory scan would be behaviour a reader has to know to look for, and defaulting
it off would silently stop serving precompressed assets for anyone who ships them — a
failure visible only as a bandwidth bill. The trade is stated at the call site instead:
call it when the root has no sidecars, and accept that one appearing later is not served.
2. **An eager content cache: measured at 2.55x nginx.** `examples/bench_cached.rs` is a
realistic *hit* path — it takes the router's segments, refuses `..`/separators/NUL and
dot-prefixed segments, hashes a key, looks the entry up, compares `If-None-Match`, and
emits the same seven headers as the real path. Interleaved with the disk path and nginx,
three reps, on this site's `styles.css`:
| | mean req/s | µs/req |
|---|---|---|
| disk, no sidecar probe | 65,134 | 15.35 |
| **cached hit** | **169,554** | **5.90** |
| nginx | 66,463 | 15.05 |
The hit costs **5.90 µs against the 5.09 µs fixed-response floor**, so every check, the
lookup and all seven headers together add **0.81 µs**. Two earlier estimates were wrong
in both directions: 3x was extrapolated from a no-lookup response, then the correction
to "1.5–2x" was too pessimistic. Measured, it is **2.6x the disk path and 2.55x nginx**,
and the worst cached run still beat the best nginx run by 2.3x.
**What that benchmark does not measure, and what would decide the design:** populating
the cache is a *second resolution path* that must be the exact inverse of the request
path — it has to prove containment per file (a naive walk follows a symlink out of the
root and caches its contents under an in-root key), refuse anything that is not a
regular file (`File::open` on a fifo blocks, so eager population can hang the boot), and
agree with the request path on a case-insensitive filesystem, where `/STYLES.CSS` and
`/Styles.Css` both serve today and would both miss an exact-key cache. The cache can
never be complete, so the disk path stays forever and every guarantee has to hold on
both. That is the cost, and it is not in the number above.
3. **`sendfile` is not the lever, and the old note claiming otherwise was wrong.** nginx
with `sendfile off` measured **faster** than with it on (73,389 vs 65,751): macOS's
`sendfile` is poor, and for a small file the setup exceeds the copy it saves. hyper's
inability to use it is costing nothing here.
### Full suite, 2026-08-17, Mac16,2 — generated fixture, sidecars present
Every earlier table in this file was measured on a fixture that **did not contain the files
it claimed to measure**. `bench/www` held only `index.html`; `styles.css` was absent, so all
four `styles.css` rows were 404 throughput reported as file-serving throughput. It was caught
only because a 14 KB file appeared *faster* than a 1.8 KB one. `faceoff.sh` now generates the
fixture (including `.br`/`.gz` sidecars) and calls `assert_200` before every measured window,
which aborts the run rather than producing a plausible number. That guard was reintroduced
against the real defect and seen to fire before being committed.
One worker each, `oha -z 8s -c 50`, three interleaved reps. index.html 1,609 B,
styles.css 14,620 B.
| `mini-static`, no probe | 58,573 | 50,433 |
| nginx, plain | 69,133 | 54,355 |
| **`mini-static`, cached** | **146,847** | **111,179** |
| `mini-static`, probing (serves `.br`) | 40,330 | 34,821 |
| nginx, `gzip_static` (serves `.gz`) | 68,748 | 53,996 |
**These nginx figures have `open_file_cache` off, which makes the cached row an unfair
comparison — see the next section. Feature-matched, the cached row is a tie, not a 2x win.**
Three results, and the middle one is not good news:
1. **Cached, we are 2.05–2.12x nginx.** No filesystem on the hit path, and it holds on both
file sizes.
2. **Uncached and not probing, nginx leads by 8–18%** — consistent with the 11–13% recorded
below, and the honest position for a root that is written to while running.
3. **Probing with sidecars actually present, nginx leads by 55–70%.** This is new, and it is
the fixture defect's silver lining: with no sidecars on disk the probe merely failed twice
and cost ~11%, so the real cost of the feature working was never measured.
#### The cached row against nginx's own equivalent feature: a tie
nginx has a nearest equivalent to the content cache — `open_file_cache`, which holds open
descriptors plus their `stat` results, so a repeat request skips `open()` and `fstat()` too.
The table above has it **off**, which is the default, and that is what produced a "2x nginx"
claim. Enabling it:
| nginx, plain | 59,580 | 57,907 |
| nginx, `open_file_cache` | 120,870 | 114,561 |
| **ours, cached** | **130,023** | **119,314** |
`open_file_cache` roughly doubles nginx and closes ~95% of the gap. Feature-matched we are
**+7.6% and +4.1% ahead, against a run-to-run spread of 4–9%** — at or below the noise floor.
An earlier 3-rep run put it at +9% and +5%, so the direction is consistent and the magnitude
is small. **The honest claim is parity with a tuned nginx, not a multiple of it.** (Absolute
numbers here are lower than the table above because the 5-rep run drifted thermally; only the
within-run ratios are comparable, which is the whole reason this harness interleaves.)
Two things follow, and the second one is a criticism of our design:
1. **We are not beating disk I/O — neither server touches the disk.** A 14 KB file lives in
the OS page cache after the first request. What both caches remove is *syscalls*, and that
is the entire ~10 µs per request the filesystem was costing.
2. **nginx pays a better price for the same speed.** `open_file_cache` holds descriptors and
metadata, so its memory is O(number of files) rather than O(bytes of content), and
`open_file_cache_valid` revalidates on a timer, so a file changed under a running server is
picked up. Ours holds full content — memory equals the size of the served root — and never
revalidates, so it demands a restart and refuses to coexist with live-reload. nginx
demonstrates that the staleness guarantee we gave up did not have to be given up to reach
this speed.
The measured RSS difference on this fixture is 64 KB against a 36 KB root, which proves
nothing except that the accounting is right; the cost is arithmetic, not empirical — a
500 MB root is 500 MB of RSS.
#### Why the probing path is 31% behind its own non-probing path
It serves *less* data — 480 bytes of brotli against 14,620 uncompressed — and is slower, so
this is syscalls, not bytes. `handle_request` opens and `fstat`s the **original** file to
prove containment, and only then probes for a sidecar; when the sidecar wins, that first
descriptor is discarded. A sidecar-served request therefore pays two `open()`+`fstat` pairs
and uses one. nginx's `gzip_static` pays one, which is why its precompressed row is
indistinguishable from its plain row while ours drops a third.
The wasted `open()` is not load-bearing: the ETag and `Last-Modified` already come from the
*sidecar's* metadata, so the original is opened purely for a containment check that the
sidecar's own descriptor could satisfy. Deferring it until the probe declines should recover
most of the gap for precompressed deployments. Not done here — it is a change to the
resolution path, which is the one place in this crate where a shortcut has already caused a
vulnerability, so it wants its own plan and its own mutation coverage rather than a
benchmark-driven patch.
### Feature-matched against nginx, interleaved and repeated
**Superseded by the table above, and measured on the broken fixture.** Kept for the two
corrections it records. Earlier figures in this file compared a probing `mini-static` against an nginx that was not
probing, and were measured non-interleaved on a machine that drifts. Both faults are fixed
here: matched configurations, alternating servers, three reps each, on real site content.
| `/index.html` (1.8 KB), neither probing | 68,411 | 76,222 | **+11.4%** |
| `/styles.css` (14 KB), neither probing | 60,348 | 68,367 | **+13.3%** |
| `/index.html`, both probing | 60,859 | 72,588 | +19.3% |
| `/styles.css`, both probing | 55,658 | 65,206 | +17.2% |
**Matched, nginx leads by 11–13%.** With precompressed support on both sides the gap widens
to 17–19%, because this crate probes twice (`.br` and `.gz`) where nginx probes once (`.gz`
only; brotli needs a third-party module). Run-to-run spread is 5–10% on both servers, so
11–13% is near the edge of what this harness can resolve — treat it as "roughly 10–15%".
#### Two earlier claims in this file were wrong
- **"~8% behind while doing strictly more" was too generous.** It came from comparing runs
taken minutes apart. Interleaved and matched, it is 11–13%.
- **"our probe is 2.6x cheaper than nginx's" was backwards.** Per probe: this crate pays
**0.90 µs** (68,411 → 60,859 req/s for two), nginx pays **0.66 µs** (76,222 → 72,588 for
one). nginx's probe is about 27% cheaper than ours, not the reverse. The earlier figure
compared a `gzip_static` run against a plain run measured at a different time, and
attributed the drift to the feature.
Both errors had the same cause, and it is the one this file already warns about: numbers
taken at different times on this machine are not comparable. Interleave, repeat, and quote
ranges.
## Criterion micro-benchmarks
```bash
cargo bench # both suites
cargo bench --bench handle_request -- --save-baseline before
# ... make a change ...
cargo bench --bench handle_request -- --baseline before
```
`handle_request` covers `small_file_200`, `large_file_200`, `not_modified_304` and
`sidecar_hit_200`. `path_resolution` covers shallow, deep, percent-decoded, traversal-
rejected and directory-index paths.
**Read the direction carefully.** `--baseline before` reports how the *current* code
compares to the saved baseline: "Performance has regressed" means the run you just did is
slower than the baseline, whichever code that was. Comparing an old checkout against a
baseline saved from new code inverts every sign, which is easy to do and easy to
misreport.
### What they cannot see
These measure two functions. They do not measure the accept loop, connection handling,
response writing, or anything a caller wraps around them — so a change to those is
invisible here and needs `throughput.sh`.
This is not hypothetical. A change that cost 5–6% on `not_modified_304` and
`sidecar_hit_200` passed the full test suite, the lint gate and the mutation suite,
because **none of those measure speed**. Only running the benchmark caught it. If a change
touches the request path at all, run both harnesses.