---
title: Benchmarks
description: Measure isolated and concurrent builds with cold and warm caches
---
## CI coverage
The [Perf gate](https://github.com/kunobi-ninja/kache/actions/workflows/perf-gate.yml)
measures **hk, eza and aube** on PRs. Each subject runs Kache from the PR
and its merge base, and mbx, sequentially on one runner. sccache runs weekly.
| Trigger | Subjects | Isolated samples per tool | Contention samples per tool |
| --- | --- | --- | --- |
| Same-repository PR, ready for review | hk, eza, aube | Kache head and base, each: 2 cold, 6 same-checkout warm, 6 cross-checkout warm. mbx: 1 cold, 1 same-checkout warm, 1 cross-checkout warm | 1 cold + 1 warm batch |
| PR with the `bench` label **or** `[bench]` in its title | hk, eza, aube | 2 cold, 6 same-checkout warm, 6 cross-checkout warm | 2 cold + 6 warm batches |
| Nightly, 03:37 Europe/Zurich | hk, eza, aube | Kache: 2 cold, 6 same-checkout warm, 6 cross-checkout warm. mbx: 1 cold, 3 same-checkout warm, 3 cross-checkout warm | 1 cold + 3 warm batches |
| Weekly, Sunday | hk, eza, aube | As the nightly, plus sccache: 1 of each | 1 cold + 3 warm batches |
| Nightly | Lance, OpenDAL | Kache and mbx, each: 1 cold, 3 same-checkout warm, 3 cross-checkout warm | — |
| Nightly | LLVM | Kache: 1 cold, 3 same-checkout warm, 3 cross-checkout warm | — |
| Nightly | Firefox Linux/Windows, SurrealDB, cuda-oxide, kobe | One measurement per configured phase | — |
The opt-in also runs on same-repository draft PRs. Adding either trigger or
pushing a new commit starts a measurement; a newer run cancels the older one.
Fork PRs cannot use the self-hosted benchmark runner.
Firefox Linux/Windows, including the next-commit variants, run only on the
nightly schedule. Manual `all` excludes Firefox. Substrate is no longer a
benchmark subject.
The two Firefox/Windows arms run in their own workflow on the same nightly
cron. Each takes five to six hours. A workflow run finishes only when its
slowest job does, so while they shared a workflow with the Linux matrix, every
other arm's telemetry waited for them.
| Subject | Kache | sccache | mbx |
| --- | --- | --- | --- |
| hk, eza, aube | yes | weekly | yes |
| Firefox Linux | yes | yes | C and C++ only |
| Firefox Windows, LLVM, cuda-oxide, kobe | yes | — | — |
| SurrealDB, Lance, OpenDAL | yes | — | yes |
The sccache arms have their own workflow and their own schedule: weekly, and
again whenever the pinned sccache version changes. That covers Firefox and the
hk, eza and aube suites. sccache is pinned in
`mise.toml`, so a nightly arm would measure the same sccache binary every night
while only Kache changed. The per-PR perf gate already compares Kache with
itself. Running it separately also keeps its 75 minutes off the nightly's
critical path, so nightly results reach the collector sooner. mbx tracks its
latest release, so it stays nightly.
The Firefox mbx arm is the exception. A Firefox build is too long for the
nightly, so it runs weekly from the same workflow. It builds with
`mbx exec ./mach build`, which caches C and C++ compiles. Cargo calls inside
that session are not cached, so the arm's Rust crates compile in every phase
and its hit rate counts C and C++ only.
hk retains its prediction-disabled and next-commit scenarios. The next-commit
Kache variants separate exhaustive prediction verification from timing.
## Cold CI prefetch qualification
The `Prefetch cold CI qualification` workflow compares three prefetch on/off
pairs using a pinned eza revision, Rust 1.90.0 and two Cargo jobs. A protected main push produces
one immutable seed containing Kache, cached artifacts, manifests and namespace
shards. Every consumer starts with empty local and target directories.
The harness sets `KACHE_EVENT_ROOT` to the workload root so Rust, build scripts
and native compiles share one session even when Cargo creates no directory tag.
Consumers read that seed through a loopback S3 fixture with a 50 ms delay before
each response. The fixture records requests and rejects writes. This measures
prefetch under controlled request latency; it does not model cloud bandwidth,
packet loss or variable service latency. Remote key listing is enabled in both
arms. The ordinary S3 default leaves listing disabled.
The collector requires matching executable hashes, drained telemetry, complete
GET-body precision accounting and matching latency controls. It reports full
GitHub job durations separately from the build phases. HTTP logs and transport
profiles are included in each consumer artifact.
Startup downloads carry no session ID. In this isolated runtime, the collector
joins them to the sole demand session only after the daemon has drained all
prefetch tasks. Multiple demand sessions or incomplete receipts fail admission.
A manual workflow run reuses the binary in its selected seed. To qualify a new
Kache revision, use a new seed produced by a main push. Reusing an old seed does
not test newly merged fixes.
For a Linux sandbox experiment before merging the workflow, build Kache from the
revision under test, install Rust 1.90.0, then run:
```sh
python3 scripts/prefetch-qualification.py experiment \
--root /tmp/prefetch-experiment \
--output /tmp/prefetch-results \
--binary "$PWD/target/release/kache" \
--revision "$(git rev-parse HEAD)" \
--response-delay-ms 50
```
Use a fresh root for each experiment. The result contains three alternating
on/off pairs and build durations. A local experiment seed is rejected by CI;
its timings are not GitHub job durations. Use `--response-delay-ms 0` for an HTTP
baseline without an injected delay.
Pass `--archive /path/to/seed.tar.gz` to repeat an experiment with the same local
seed. Its recorded revision must match `--revision`; the bundled binary is used.
## Phases
| Phase in JSON | Cache before the build | Build directory |
| --- | --- | --- |
| `cold` | Empty | Empty |
| `warm_same_tree` | Snapshot populated by cold | Empty, original checkout |
| `warm` | Same cold snapshot | Empty, different checkout at the same revision |
| `pull` | Populated at commit A | Empty, original checkout moved to commit B |
hk and eza run the first three phases through all three tools. Other clone
scenarios run cold and cross-checkout warm. The separate `pull` scenarios
measure cold followed by the next-commit rebuild; they do not measure an
incremental build with existing artifacts.
Cold uses the cache tool with an empty store. It is not an uncached build
without a wrapper. The operating system's page cache is not flushed.
The short suite fetches dependencies before timing. It alternates arm order,
keeps each arm's store and paths separate, and measures a new cold seed every
third sample. The other samples run with `--retry`, which restores the saved
cold seed instead of building it. That reused cold value is left out of the
cold sample count and duration telemetry. Each warm phase restores
that seed and clears its build artifacts before starting.
## Reading a PR result
The PR comment opens with one table: Kache base and head for each subject, as
median times per build with the paired change. Timings for every tool,
contention counters, disk use and the paired test sit in folded sections below
it. With more than one sample, each time shows its range.
`summary.json` carries the statistics; `samples.json` retains raw measurements,
source/tool versions, arm order, and the head/base/instrument identities.
Build logs and engine reports are retained for each sample.
Only Kache head versus base decides the PR verdict. Increased warm misses or
passthroughs fail independently of elapsed time. Timing tests compare against
zero change; there is no automatic percentage or duration allowance.
Before measuring, the harness records a randomized base/head order for each
cold-cache seed. Even numbers of seeds use balanced AB/BA assignments; odd
numbers use independent assignments. The same order holds for that seed's warm
repeats. Those repeats form one statistical block. The report shows geometric
paired changes, exact two-sided randomization probabilities and simultaneous
95% family intervals. Intervals assume a constant proportional effect and
condition on this run's measurement slots. They do not describe variation
across machines or nights.
Declare the timing and cold-count family before running with
`--comparison-family-size`. The three-subject PR suite declares 18 comparisons
and uses Bonferroni correction, which allows dependence between phases.
A detected positive change fails; a detected negative change is an improvement.
Other results are **inconclusive**, including legacy data without a recorded
randomization. Nonsignificance does not establish parity.
The default six isolated samples reuse two cold seeds. They provide only two
randomized blocks, so timing remains inconclusive. Six fresh contention seeds
also cannot resolve a two-sided test at this suite's corrected level. Sample
counts and runtime remain unchanged. For a focused comparison, use
`--samples 12 --cold-every 1 --skip-contention`; predeclare a larger family when
combining workloads. Repeat on another lease before generalizing the result.
Other tools provide context and never decide the verdict.
The method follows [randomized blocks](https://www.itl.nist.gov/div898/handbook/pri/section3/pri332.htm),
[exact paired permutation tests](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.permutation_test.html)
and [Bonferroni intervals](https://www.itl.nist.gov/div898/handbook/prc/section4/prc463.htm).
A successful measurement job means its validity and regression checks passed;
an **INCONCLUSIVE** report is not a performance approval.
Missing phases, zero restores, cache errors, or invalid Kache verdicts produce
**INVALID MEASUREMENT**. A fast build that did not exercise the cache is not a
performance result. Competitor timings provide context and never fail a PR
because another tool is faster.
The engine, scenarios, installer, and comparison policy come from the default
branch. Only the two Kache binaries vary with the PR. Changes to the measuring
instrument therefore take effect after they merge. `perf-gate/warm` remains an
optional status; this workflow does not change branch protection.
## Local commands
```bash
just bench # list Kache scenarios
just bench hk --warm-same-tree
just bench eza --warm-same-tree
just bench-pr # one Kache side of hk and eza
just bench-short hk # three tools, six warm samples each
just bench-short eza
just bench-sccache hk --warm-same-tree
just bench-sccache eza --warm-same-tree
just bench-mbx hk --warm-same-tree
just bench-mbx eza --warm-same-tree
just bench-mbx hk-pull
just perf-gate-local # HEAD against its merge base, with the gate's verdict
```
The complete short suite runs on Linux. It measures mbx and sccache only when given `--mbx <binary>` or `--sccache <binary>`.
On macOS, pass `--skip-contention` to run its isolated builds. It refuses to overwrite its
output directory; pass `--output <fresh-path>` after the sample count for another
run. Run builds on an otherwise quiet host. A CI job takes much longer than
its build phases: tool installation, compiling Kache, preparing the subject,
and repetitions all add to its elapsed time.
Individual benchmark runs keep reports under `tmp/bench/<scenario>/` and archive
completed runs in `runs/`. The short suite keeps `samples.json`, `summary.json`,
`perf-gate.md`, `metrics.otlp.json`, and per-sample `logs/` in its output directory.
See the [scenario format](https://github.com/kunobi-ninja/kache/tree/main/scenarios#readme)
when adding a subject. For choosing a cache, see [Kache or sccache?](/docs/getting-started/comparison).
## Contention on one runner
`bench-contention.py` runs six Cargo commands for hk or eza: check, Clippy,
and test compilation, each with default features and a second feature set.
hk uses all features/targets; eza uses no default features and all targets.
Every command owns a separate checkout and target directory. Commands share
one store of the selected tool within an arm; arms and cold seeds have separate stores. Cold
starts empty. Warm clears all six targets and restores the completed cold store.
Both phases use the debug profile without Cargo incremental state. These are
batch measurements; eza's isolated release build remains a separate case.
```bash
just bench-contention --project hk --samples 3 --jobs-per-build 4 \
--output /tmp/hk-contention
# Compare two release binaries with identical settings and alternating order.
python3 scripts/bench-contention.py --project hk --samples 3 \
--arm base=/path/to/base/kache,1 --arm head=/path/to/head/kache,1 \
--jobs-per-build 4 --output /tmp/hk-contention-comparison
```
eza's `--all-features` graph enables vendored OpenSSL and exceeds the short
case's budget. `--full-features` retains that stress workload for standalone
runs and exports it as `bench-eza-contention-full-features`. On the validation
runner, sccache 0.15.0 failed OpenSSL's assembler generation in that variant;
those results are invalid and excluded from the normal comparison.
The number after the comma in each `--arm` value enables (`1`) or disables (`0`)
Kache's scheduler.
Use `--parallelism 1`, `3`, or `6` to vary overlap while retaining the same six
commands and independent targets. `--jobs-per-build` sets Cargo's parallelism
inside each command; it does not change the scheduler's permit pool.
Run this Linux-only benchmark on a quiet runner. The report retains CPU
affinity, cgroup CPU/memory limits, CPU throttling, sampled real compiler counts,
and cgroup memory usage, including filesystem page cache. Compiler peaks exclude wrapper processes and include
native compiler/preprocessor frontends. Sampling can miss short-lived processes.
Cgroup measurements include other work inside that container, so compiling
Kache or running tests alongside the measurement invalidates comparisons.
The export includes `samples.json`, `summary.json`, `report.md` and
`metrics.otlp.json`. Each phase adds its Cargo logs, raw Kache events, mbx
session streams, Perfetto traces, and process samples in `result.json`. The
export records
binary hashes, toolchain, source revision, and instrument hash. It reports
compiler invocations, duplicate compilations of a cache key, and time spent
computing keys, waiting for peers, and waiting for permits. Summed wait times
overlap; they are not elapsed-time savings. Event logs use a larger retention
budget for the six-job batch, and detected rotation invalidates the measurement.
After each cold and warm batch, the runner measures storage outside the timer.
`samples.json` and OTLP include logical bytes, bytes counted once per inode,
and allocated blocks for the cache, targets, and their combined footprint.
Cache components distinguish blobs, the SQLite index, mbx session logs, and
other metadata. The report shows mean allocated GiB. Hardlinks count once;
shared reflink extents are unresolved, so allocated blocks are not exclusive
physical disk usage. Sources, Cargo home, runtime logs, and cold snapshots
are excluded. Cache-resident session logs remain included.
`bench-short` includes contention for hk and eza in PRs and nightly. The same
three tools run identical commands; Kache head and base also run on PRs.
Contention appears as its own report section and as `bench-hk-contention` or
`bench-eza-contention` in the existing OTLP metrics. The suite archives raw
results under `contention/` and includes the statistics in `summary.json`.
Missing backend reports or failed builds invalidate the suite.
PRs measure six fresh cold contention seeds and six warm batches per Kache
arm. Cold duplicate counts use the recorded order randomization and the same
predeclared family as timings. Warm increases in duplicate compilations,
misses or passthroughs fail immediately. Nightly runs without a base arm report
descriptive counts and timings.
Use `--contention-samples 12 --contention-cold-every 1` with `bench-short`
for twelve fresh paired seeds in a focused comparison,
or `--skip-contention` to focus on isolated builds. To run only contention
with all three tools, add `--sccache /path/to/sccache --mbx /path/to/mbx` to the
standalone command. Compiler invocation and internal wait counters come from
Kache events. Other tools report their own cache statistics, and fields they
do not report show as unknown. mbx's `wrapper_phases_ns` holds exclusive phase
durations, while Kache's key timer includes dependency discovery, so the two
timers measure different work.
### Detailed phase tracing
For a diagnostic run, add `--trace-phases` to `bench-contention.py`. Kache
then records real monotonic intervals under each phase's `kache-phases/`
directory and merges them into `kache.trace.json`, which Perfetto can open.
The trace marks prediction decisions and measures prediction reads and
validation, dependency discovery, file hashing, scheduler waits, compiler
execution, local lookups, and Rust artifact restoration. Nested spans include
their children; summing them double-counts that work.
Outside the benchmark, set `KACHE_PHASE_TRACE_DIR=/tmp/kache-phases` when
running Cargo through Kache. Each wrapper writes its own trace on return.
The trace covers the wrapper's calling thread only: it leaves out CLI startup
and work in child processes or other threads. Trace writing adds overhead, so
compare performance on untraced runs. The benchmark marks traced runs
as diagnostic and rejects missing or truncated traces.
`kache stats --format perfetto` remains the compact event-based view: its
phase offsets are reconstructed from millisecond totals. The opt-in files
above record actual interval placement and retain sub-millisecond work.