# Quality installation benchmark
This benchmark compares bot-forge scheduling with conventional serial installation on the same
macOS host. It uses the machine's current Rust/Cargo 1.89.0 toolchain and never modifies or removes
that installation.
## Compared targets
The measured target set is identical on both sides:
- Rust 1.89 components: `rustfmt`, `clippy`, `llvm-tools-preview`
- Cargo tools: `cargo-expand 1.0.126`, `cargo-nextest 0.9.120`,
`cargo-audit 0.22.2`, `cargo-deny 0.20.2`, `cargo-geiger 0.13.0`,
`cargo-llvm-cov 0.9.0`, `bot-gate 1.2.1`, `bot-metric 1.2.0`
- All measured Cargo tools are fetched from crates.io
The host's Git, Clang, SDK, certificates, `pkg-config`, and OpenSSL are shared prerequisites. The
three Rust components are already present in the current toolchain: each method checks them inside
its end-to-end timer but does not uninstall and redownload them. The eight Cargo tools are the clean
installation workload.
## Isolation and fairness
Every sample receives new `HOME`, an empty `CARGO_HOME`, `BOT_FORGE_HOME`, managed bin, and
installation directories. `RUSTUP_HOME` points at the current machine's toolchain. The benchmark
path excludes the user's Cargo and bot-forge bins, so already installed Cargo tools cannot satisfy
detection. No registry index, crate archive, Git object, target output, or bot-forge artifact is
copied into a sample.
Both methods replace the `crates-io` source with the same sparse rsproxy endpoint, so both the
index and crate downloads come from rsproxy:
```text
sparse+https://rsproxy.cn/index/
```
After each sample, the runner also checks Cargo's cached registry `config.json` and rejects the
sample unless every observed crate download endpoint belongs to `rsproxy.cn`.
Both methods use all logical CPUs reported by the host:
- bot-forge chooses concurrent Cargo builds and their inner Cargo jobs dynamically.
- the serial baseline runs exactly one install command at a time and gives that command all CPUs
through `CARGO_BUILD_JOBS`.
The runner alternates method order between rounds. Use at least four rounds for a publishable
number because empty Cargo homes deliberately retain real network variance.
## Run
```bash
python3 benchmarks/quality-install/bench.py --runs 4
```
For a quick functional smoke run:
```bash
python3 benchmarks/quality-install/bench.py --runs 1
```
Reports and per-sample logs are written under `target/benchmarks/quality-install/`. Successful
benchmark homes are removed automatically. A failed run prints and retains its work directory for
diagnosis; `--keep-work` also retains successful homes.
The reported speedup is `serial median / bot-forge median`. It is an end-to-end comparison for the
11 requested targets: three current-toolchain detections plus eight clean Cargo installations. It
is not a compiler microbenchmark.