---
title: Benchmarks and reports
description: Measure cold, local-hit, remote-hit, and cross-checkout behavior
---
The repository benchmark harness builds real projects in separate checkouts. This tests wall time and whether path normalization produces portable keys.
## Nightly benchmark matrix
The [Bench workflow](https://github.com/kunobi-ninja/kache/actions/workflows/bench.yml) is scheduled every night at 03:37 Zurich time (01:37 UTC in summer, 02:37 UTC in winter). It runs:
- Firefox cold/warm builds on Linux and Windows
- LLVM, Substrate, SurrealDB, Lance, and OpenDAL cold/warm builds
- Firefox through sccache for comparison
- Firefox source-update builds on Linux and Windows
- hk cold, warm and next-commit builds on Linux: a mid-size Rust CLI whose build scripts compile C and assembly through cc-rs, wired the way `kache init` configures a machine
- hk with input predictions off, as the control for the hk arm's warm phases
- The hk, Lance, OpenDAL, Substrate and SurrealDB builds through mbx at its latest release, for side-by-side comparisons
- eza cold, warm and cross-worktree builds on Linux: the smallest subject in the matrix and the only one built through a fat-LTO release profile, with its bundled libgit2 and zlib compiled through the same cc-rs wiring
Jobs upload their reports, Perfetto traces, and logs even when a benchmark fails. GitHub retains those artifacts for 30 days.
A scheduled run is not automatically a valid result. Check that the individual job succeeded and that its benchmark verdict is `ok` before using its timing or hit-rate numbers.
## Per-pull-request perf gate
The nightly matrix reports absolute numbers hours after a change lands. The [Perf gate workflow](https://github.com/kunobi-ninja/kache/actions/workflows/perf-gate.yml) answers a narrower question at review time: is this pull request slower than the commit it is a delta against?
It measures one mid-size subject — `bench-pr-cargo`, the `rust-lang/cargo` tree pinned by commit — in three phases per side:
| phase | store | build tree |
| --- | --- | --- |
| cold | empty | fresh |
| warm | populated by cold | fresh, same path |
| cross-worktree | populated by cold | fresh, different path |
The pull request head and its merge base are both measured, back to back on the same runner, so the report is a delta rather than a comparison against a stored figure from another machine. A warm build more than 5% slower than the merge base fails the gate. Cold and cross-worktree deltas are reported without failing while their noise floor is still being established. The numbers land as a comment on the pull request.
The engine times every phase in milliseconds (`wall_ms` in the result JSON) and the gate takes its delta on that field. The subject's warm build runs about 15 seconds on the gate's runner, where a whole-second tick would already be 7% of the build, more than the threshold. The whole-second `wall_s` next to it is derived from `wall_ms` and stays in the result JSON for older readers. Speedups and the nightly telemetry are computed from `wall_ms`.
The gate refuses to compare a warm build shorter than 5 seconds. Below that, 5% of the build is within the fixed cost of spawning the shell, cargo, and its dependency resolution on a shared runner, so a delta would measure the runner rather than kache. The refusal is reported as `INVALID MEASUREMENT`, distinct from a failing delta.
Only the two `kache` binaries come from the pull request. The benchmark engine, the pinned subject, and the threshold all come from the default branch, so both sides are measured with the same instrument. One consequence to expect: a pull request that changes the engine or the scenario is not measured by its own version of them, and those changes take effect for the pull requests that follow.
The gate runs only for branches in this repository: the benchmark needs a self-hosted runner, and fork code never touches one. It is not a required check.
## Run a scenario
```bash
just bench # list Kache scenarios
just bench firefox
just bench substrate
just bench lance
just bench opendal
just bench eza
just bench llvm
just bench-retry firefox # reuse the saved cold state
just bench-trace firefox # include key-diff artifacts
just bench-pr # one side of the per-PR perf gate
just bench-sccache
just bench-sccache firefox
```
Firefox is a large mixed Rust/C++ workload and needs about 50 GB of scratch space. Run benchmark builds sequentially on an otherwise quiet host.
## Trust the verdict first
Each summary has a `verdict`. Use timing and hit-rate results only when it is `ok`. A degraded verdict can mean checkout-path leakage, an unexpected miss, or inconsistent cache placement.
Compare medians from several runs. Keep repository revision, toolchain, machine, target, profile, and remote location fixed.
## Output
Latest artifacts are under `tmp/bench/<scenario>/`:
- cold and warm JSON/Markdown reports
- Perfetto-compatible trace files
- build and wrapper logs
- scenario summary with tool version
- optional key-diff JSON and Markdown
- sccache reports for comparison runs
Completed runs are archived below `tmp/bench/<scenario>/runs/` so later runs do not erase the evidence.
## Reports outside the harness
```bash
kache report --format text --since 24h
kache report --format json --output report.json
kache report --format trace --output trace.json
kache report --format markdown --root "$PWD"
```
`trace`, `perfetto`, and `chrome-trace` produce the same trace format. Open it in Perfetto or Chrome's trace viewer.
## Add a workload
Benchmark definitions live under `scenarios/bench-*`. A scenario declares its repository, revision, setup, wrapper wiring, build command, and optional patches. Start from `bench-firefox`, `bench-substrate`, or `bench-llvm` and read the [scenario format](https://github.com/kunobi-ninja/kache/tree/main/scenarios#readme).
For choosing a cache before benchmarking, see [Kache or sccache?](/docs/getting-started/comparison).