# Weir Performance Budget Policy
## Overview
This project maintains a **machine-readable performance budget** to prevent
unintentional performance regressions. Every benchmark has a declared budget
derived from its baseline median. CI checks enforce these budgets on every
relevant change.
---
## How Benchmarks Are Run
Benchmarks live in `benches/` and use [Criterion.rs](https://bheisler.github.io/criterion.rs/book/).
```bash
# Run all benchmarks
cargo bench
# Run a specific benchmark
cargo bench csr_normalize/tiny/10
# Run benchmarks required for budget check (includes cpu-parity benches)
cargo bench --
```
Some benchmark binaries require the `cpu-parity` feature. The default feature
set includes `test-harness`, which in turn enables `cpu-parity`, so a plain
`cargo bench` is sufficient in most cases.
---
## How Budgets Are Calculated
Each budget is computed from the **Round 0 baseline** stored in
`benches/baselines/`:
```
budget_ns = round(baseline_median_ns × 1.15)
```
- **baseline_median_ns**, the Criterion `median.point_estimate` from the
`round0` baseline run.
- **1.15 multiplier**: a 15 % regression allowance (see rationale below).
- **round()** (budgets are stored as integers for simplicity).
The generated manifest is at **`benches/budgets.json`**.
---
## The 15 % Regression Threshold Rationale
- **Noise floor**: Criterion already performs statistical filtering, but CI
runners still exhibit run-to-run variance (thermal throttling, noisy
neighbors, etc.). A 15 % headroom absorbs this without masking real
regressions.
- **Safety margin for acceptable trade-offs**. A 10 % improvement in
maintainability or correctness that costs 5 % performance should not block
a PR.
- **Calibrated to our data**: Round 0 baselines show typical MADs well
under 5 %; 15 % is roughly 3× that spread, giving a strong statistical
signal when the threshold is breached.
If a deliberate change is expected to exceed a budget, the baseline must be
re-established (see below).
---
## Updating Baselines
When a performance-justified change lands, update the baseline so future
comparisons are apples-to-apples:
```bash
# Re-run all benchmarks and overwrite the 'round0' baseline
cargo bench -- --save-baseline round0
# Then regenerate budgets.json
python3 -c "
import json, glob
from pathlib import Path
baselines = Path('benches/baselines')
budgets = {}
for est in sorted(baselines.rglob('round0/estimates.json')):
bench = json.load(open(est.parent / 'benchmark.json'))
median = json.load(open(est))['median']['point_estimate']
gid, fid, vs = bench['group_id'], bench['function_id'], bench['value_str']
cfg = f'{fid}/{vs}' if fid and vs else (fid or vs or gid)
prefix = f'{gid}/'
cfg = cfg if not str(cfg).startswith(prefix) else str(cfg)[len(prefix):]
budgets.setdefault(gid, {})[cfg] = {
'budget_ns': round(median * 1.15),
'baseline_ns': round(median),
'full_id': bench['full_id']
}
json.dump(budgets, open('benches/budgets.json', 'w'), indent=2)
"
```
> **Tip:** Commit both the updated `benches/baselines/` files *and* the
> regenerated `benches/budgets.json` in the same PR so the history is atomic.
---
## Checking Budgets in CI
Run the budget checker script:
```bash
# Full check (runs all benchmarks, may be slow)
python3 benches/check_budgets.py
# Quick CI-friendly check (reduced sample size / measurement time)
python3 benches/check_budgets.py --quick
# Check a single benchmark or family
python3 benches/check_budgets.py --bench-filter csr_normalize
# Check against existing results without re-running
python3 benches/check_budgets.py --no-run
```
The script exits with code `0` when all budgets are satisfied and `1` when any
benchmark exceeds its budget or is missing.
### Example GitHub Actions snippet
```yaml
- name: Check performance budgets
run: python3 benches/check_budgets.py --quick
```
---
## Adding New Benchmarks
1. **Create the benchmark source** in `benches/<name>.rs`.
2. **Register it** in `Cargo.toml`:
```toml
[[bench]]
name = "my_new_bench"
harness = false
```
3. **Run it once** to generate the baseline:
```bash
cargo bench my_new_bench -- --save-baseline round0
```
4. **Regenerate `benches/budgets.json`** (see "Updating Baselines" above).
5. **Commit** the new baseline directory and the updated `budgets.json`.
---
## Files
| `benches/baselines/` | Criterion baseline data (`round0/` for each benchmark) |
| `benches/budgets.json` | Machine-readable budget manifest |
| `benches/check_budgets.py` | CI script that enforces budgets |
| `BENCH_POLICY.md` | This document |