# Reference Audio Benchmarks
This directory contains a benchmark system for comparing time-stretch quality against professional DJ software (Ableton Live, Serato, Traktor, etc.).
## Directory Layout
```
benchmarks/
manifest.toml # Track + reference descriptions (checked into git)
README.md # This file
audio/ # NOT checked into git
originals/ # Place original WAV files here
references/ # Place professionally-stretched WAVs here
bpm-corpus/ # Place BPM-accuracy corpus tracks here (wav/mp3/aiff)
public-corpus/ # Redistributable CC corpus (scripts/fetch_public_corpus.sh)
output/ # Library output written during benchmark runs
```
## Setup
1. Create the audio directories:
```bash
mkdir -p benchmarks/audio/originals benchmarks/audio/references
```
2. Place your original WAV files in `benchmarks/audio/originals/`.
3. Time-stretch each original in your professional software of choice and save the result to `benchmarks/audio/references/`.
4. Edit `benchmarks/manifest.toml` to describe your tracks and references. Each track entry needs:
- `id` - A short identifier (used in filenames)
- `description` - Human-readable description
- `original` - Path relative to `benchmarks/audio/` (e.g., `originals/my-track.wav`)
- `original_sha256` - SHA-256 of the original WAV (for corpus lock)
- `bpm` - Original tempo in BPM
Each reference entry needs:
- `file` - Path relative to `benchmarks/audio/`
- `file_sha256` - SHA-256 of the reference WAV (for corpus lock)
- `target_bpm` - Target tempo the reference was stretched to
- `software` - Software used (e.g., "Ableton Live 11")
- `algorithm` - Algorithm/mode used (e.g., "Complex Pro")
## BPM Accuracy Corpus
The BPM accuracy harness (`qa/bpm_accuracy.rs`) scores tempo detection against
every manifest track with a `bpm` field. To grow the corpus beyond the
reference tracks, add `bpm_only = true` entries — these carry no stretched
references, may be `.wav`, `.mp3`, or `.aiff`, and are ignored by the
reference quality benchmark:
1. Drop the audio file under `benchmarks/audio/bpm-corpus/`.
2. Compute its hash: `shasum -a 256 benchmarks/audio/bpm-corpus/<file>`.
3. Add a manifest entry:
```toml
[[track]]
id = "dark-techno-01"
description = "Dark techno, 140 BPM"
original = "bpm-corpus/dark_techno_01.mp3"
original_sha256 = "<sha256 of audio file>"
bpm = 140.0
bpm_only = true
```
4. Run the harness:
```bash
cargo test --features qa-harnesses --release --test bpm_accuracy -- --nocapture
```
Per-track `METRIC` lines and a `SUMMARY` line (acc1 = % within ±2%,
acc2 = % allowing octave errors) are printed, and a JSON report is written to
`target/bpm_accuracy_report.json` for diffing between detector changes. See
`qa/README.md` for the harness's environment variables.
## Running Benchmarks
```bash
# M0 baseline command (strict validation + archive)
./benchmarks/run_m0_baseline.sh
```
This command:
- Enables strict validation (`TIMESTRETCH_STRICT_REFERENCE_BENCHMARK=1`)
- Uses a fixed 30-second analysis window (`TIMESTRETCH_REFERENCE_MAX_SECONDS=30`)
- Fails on missing files, invalid paths, or checksum mismatches
- Processes each original with all 5 EDM presets
- Prints timing/spectral/transient/loudness/length metrics
- Writes JSON metrics to `benchmarks/audio/output/report.json`
- Archives the baseline to `benchmarks/baselines/`
For ad-hoc runs without strict enforcement:
```bash
cargo test --features qa-harnesses --test reference_quality -- --nocapture
```
You can optionally set `TIMESTRETCH_REFERENCE_MAX_SECONDS=<seconds>` for deterministic
short-window analysis during ad-hoc runs as well.
### External Rubber Band Scenario
For a focused one-off benchmark against a Rubber Band render (black-box reference),
provide paths through environment variables:
```bash
TIMESTRETCH_RUBBERBAND_ORIGINAL_WAV=benchmarks/audio/originals/my-loop.wav \
TIMESTRETCH_RUBBERBAND_REFERENCE_WAV=benchmarks/audio/references/my-loop_rubberband.wav \
TIMESTRETCH_RUBBERBAND_RATIO=1.024 \
cargo test --features qa-harnesses --test rubberband_comparison -- --nocapture
```
Optional variables:
- `TIMESTRETCH_RUBBERBAND_SOURCE_BPM` + `TIMESTRETCH_RUBBERBAND_TARGET_BPM` (used if `..._RATIO` is not set)
- `TIMESTRETCH_RUBBERBAND_MAX_SECONDS` (default `20.0`, trims analysis window for determinism)
Outputs are written to `target/rubberband_benchmark/`:
- `rubberband_comparison_report.csv`
- `timestretch_vs_rubberband_output.wav`
## CI Quality Gate Subset
CI enforces a corpus-independent benchmark subset via:
```bash
cargo test --features qa-harnesses --test quality_gates -- --nocapture
```
CI also enforces a strict worst-case callback-time budget gate in release mode:
```bash
TIMESTRETCH_STRICT_CALLBACK_BUDGET=1 cargo test --features qa-harnesses --release --test quality_gates -- --nocapture
```
To emit machine-readable dashboard artifacts locally:
```bash
TIMESTRETCH_QUALITY_DASHBOARD_DIR=target/quality_dashboard cargo test --features qa-harnesses --test quality_gates -- --nocapture
```
This writes per-gate CSV files under `target/quality_dashboard/`.
This test uses synthetic DJ-like material and fails on regressions for:
- duration error
- transient alignment
- cross-correlation timing coherence
- loudness deviation
- spectral similarity by band
## Synthetic Quality Harness (Including Pitch/Formant)
Run the full synthetic benchmark harness (ignored by default because it is slower):
```bash
cargo test --features qa-harnesses --test quality_benchmark -- --ignored --nocapture
```
Outputs are written to `target/quality_benchmark/`:
- `quality_report.csv` (baseline PV vs overhauled hybrid time-stretch metrics)
- `pitch_formant_report.csv` (pitch-shift metrics for `EnvelopePreset::Off` vs `EnvelopePreset::Vocal`)
- `pitch_formant_delta.csv` (vocal-minus-off deltas per pitch factor)
- `*.wav` and `spectrogram_*.csv` artifacts for manual listening/visual inspection
## Metrics
| Length accuracy | Sample count deviation from reference |
| RMS difference | Overall loudness comparison in dB |
| Spectral similarity | STFT cosine similarity (0.0-1.0) |
| Band spectral similarity | Per-band STFT cosine similarity (sub-bass/low/mid/high) |
| Transient match rate | Percentage of reference onsets matched within 10ms |
| Cross-correlation | Peak correlation value and timing offset |
## Notes
- Audio files are gitignored. Each contributor must supply their own copies.
- The benchmark uses copyrighted audio that cannot be distributed.
- Output files are written to `benchmarks/audio/output/` and also gitignored.
## Public Corpus (redistributable)
The public corpus (ROADMAP Stage 7) is a set of Creative Commons netlabel
club tracks hosted on the Internet Archive — every license permits
redistribution and derivative works (CC0 / CC-BY / CC-BY-SA), so stretched
renders and CI artifacts built from it are safe to publish with
attribution. Fetch it with:
```bash
scripts/fetch_public_corpus.sh
```
Every file is verified against a pinned SHA-256. The scored entries carry
tempo ground truth confirmed by two independent estimators (the repo
detector and a coherent onset-envelope Goertzel scan; all scored tracks
peak at an exact integer BPM, consistent across four 60 s windows).
Entries with `bpm = 0.0` are rhythmically ambiguous (swung UK garage,
broken techno) and await ear verification — the BPM harness skips them,
but they remain valuable listening material.
CI fetches this corpus (cached) and runs `qa/bpm_accuracy.rs` against it
on every push — see the `public-corpus` job in `.github/workflows/ci.yml`.
The tracks in `bpm-corpus/` are commercial purchases for local runs only
and are never fetched or redistributed.