captchaforge 0.2.21

# Benchmarking Guide

## Quick Start

```bash
cd bench/

# Fast smoke test (~30s)
cargo run -- --suite anti-detection

# Detection speed across all fixtures (~2 min)
cargo run -- --suite detection --instances 1 --max-concurrent 4

# Full benchmark suite (~20 min)
cargo run -- --suite all --instances 2 --max-concurrent 8

# Regression comparison
cargo run -- --suite regression --baseline bench-results-report.json --strict
```

## Output Files

Every run produces 6 files:

| File | Format | Use Case |
|------|--------|----------|
| `{prefix}-report.json` | JSON | Full raw data + aggregates |
| `{prefix}-aggregates.csv` | CSV | Summary statistics per suite×fixture |
| `{prefix}-observations.csv` | CSV | Per-iteration raw data |
| `{prefix}-report.md` | Markdown | GitHub PR comment / human review |
| `{prefix}-report.html` | HTML | Self-contained interactive report |
| `{prefix}-report.sarif` | SARIF | GitHub Security tab integration |

## Interpreting Results

### Summary Table Columns

| Column | Meaning |
|--------|---------|
| `ITERS` | Total iterations (success + failure) |
| `SUCC%` | Percentage of successful iterations |
| `MED(ms)` | Median elapsed time for successes |
| `STDEV` | Standard deviation of elapsed times |
| `CV` | Coefficient of variation (stddev / mean) |

### Reading Regression Analysis

```
--- Regression Analysis ---
  turnstile/visible: +23.5% (45ms → 56ms) 🔴 REGRESSED
  recaptcha/v3:      -5.2%  (120ms → 114ms) 🟢 OK
```

A regression is flagged when:
1. Delta > `--regression-threshold` (default: 20%)
2. AND statistical significance is detected (future: Mann-Whitney U test)

## Adding a New Fixture

1. Add `pub const FIXTURE_NAME: &str` to `bench/src/fixtures.rs`
2. Register in `bench/src/server.rs::fixtures()`
3. Rules:
   - Fully self-contained (no external CDN)
   - Include `<meta viewport>`
   - Realistic DOM for `detect()` to identify
   - Inline styles only

## Adding a New Suite

1. Create `bench/src/suites/my_suite.rs`
2. Export in `bench/src/suites/mod.rs`
3. Wire dispatch in `bench/src/main.rs`
4. Signature:
   ```rust
   pub async fn run(
       report: Arc<Mutex<Report>>,
       cfg: &HarnessConfig,
   ) -> anyhow::Result<()>
   ```

## Fixture Catalog

See `bench/README.md` for the full catalog of 35+ fixtures organized by category.