# Detection benchmark
Generated by `src/scrub/benchmark.rs` for redacted 1.1.0. Do not edit by hand.
The tool reduces exposure to secrets in tool output. It never prevents it.
## What is measured and how
- The scrub engine (`Scrubber`) runs in process, not through the hook binary.
- Two configurations: default (vendor signatures, heuristic off) and heuristic on. No learned secrets in either.
- The PII rows (`email`, `phone`) are measured in a third configuration with both `pii` toggles on, and are left out of the secret totals.
- Each built-in pattern gets one fake value per placement, generated at run time by `src/fake_secrets.rs`. Keyed patterns are planted with their key, since the key is part of the match.
- Each value is planted mid-file in a `corpus/clean/*.txt` file. Files rotate across rows.
- Caught means at least one new redaction and no 8-character run of the value left outside a redaction marker.
- Seed 1792136801948421001. Replay with `REDACTED_BENCHMARK_SEED=1792136801948421001`.
## Misses
| JSON nested as escaped string | `private_key_truncated`, `gcp_sa_key_id`, `gcp_sa_private_key` | 3 | 3 | 0 |
| URL path segment | `email` (PII on) | 0 | 0 | 1 |
| mid-token | `twilio_api_key`, `twilio_account_sid`, `credentialed_url`, `openai_classic_key`, `auth_header` | 5 | 5 | 0 |
Expected-miss placements (base64, line-wrapped) appear in the recall table only.
## Recall by placement
| env line | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| shell export | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| JSON value | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| JSON nested as escaped string | harder | 43/46 (93.5%) | 43/46 (93.5%) |
| URL query parameter | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| URL path segment | harder | 46/46 (100.0%) | 46/46 (100.0%) |
| git diff added line | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| log line | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| YAML value | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| markdown inline code | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| quoted in error message | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| CSV field | plain | 46/46 (100.0%) | 46/46 (100.0%) |
| mid-token | harder | 41/46 (89.1%) | 41/46 (89.1%) |
| trailing period | harder | 46/46 (100.0%) | 46/46 (100.0%) |
| trailing comma | harder | 46/46 (100.0%) | 46/46 (100.0%) |
| base64 of the line | expected miss | 0/46 (0.0%) | 0/46 (0.0%) |
| line-wrapped at 40 | expected miss | 5/46 (10.9%) | 9/46 (19.6%) |
| all plain placements | plain | 460/460 (100.0%) | 460/460 (100.0%) |
| all placements | all | 687/782 (87.9%) | 691/782 (88.4%) |
## Opt-in PII rows (both toggles on)
| env line | plain | 2/2 (100.0%) |
| shell export | plain | 2/2 (100.0%) |
| JSON value | plain | 2/2 (100.0%) |
| JSON nested as escaped string | harder | 2/2 (100.0%) |
| URL query parameter | plain | 2/2 (100.0%) |
| URL path segment | harder | 1/2 (50.0%) |
| git diff added line | plain | 2/2 (100.0%) |
| log line | plain | 2/2 (100.0%) |
| YAML value | plain | 2/2 (100.0%) |
| markdown inline code | plain | 2/2 (100.0%) |
| quoted in error message | plain | 2/2 (100.0%) |
| CSV field | plain | 2/2 (100.0%) |
| mid-token | harder | 2/2 (100.0%) |
| trailing period | harder | 2/2 (100.0%) |
| trailing comma | harder | 2/2 (100.0%) |
| base64 of the line | expected miss | 0/2 (0.0%) |
| line-wrapped at 40 | expected miss | 2/2 (100.0%) |
| all plain placements | plain | 20/20 (100.0%) |
| all placements | all | 31/34 (91.2%) |
Excludes the expected-miss placements (base64, line-wrapped).
| `email` | 14/15 (93.3%) |
| `phone` | 15/15 (100.0%) |
## Recall by pattern
Excludes the expected-miss placements (base64, line-wrapped).
| `twilio_api_key` | 14/15 (93.3%) | 14/15 (93.3%) |
| `twilio_account_sid` | 14/15 (93.3%) | 14/15 (93.3%) |
| `private_key_truncated` | 14/15 (93.3%) | 14/15 (93.3%) |
| `credentialed_url` | 14/15 (93.3%) | 14/15 (93.3%) |
| `openai_classic_key` | 14/15 (93.3%) | 14/15 (93.3%) |
| `gcp_sa_key_id` | 14/15 (93.3%) | 14/15 (93.3%) |
| `auth_header` | 14/15 (93.3%) | 14/15 (93.3%) |
| `gcp_sa_private_key` | 14/15 (93.3%) | 14/15 (93.3%) |
All other patterns: 100% in both configurations.
## Precision on the clean corpus
Redactions in files that hold no secret. Zero is correct.
| `claude_transcript.txt` | 0 | 0 |
| `command_output.txt` | 0 | 0 |
| `docs_readme.txt` | 0 | 0 |
| `env_example.txt` | 0 | 0 |
| `gh_output.txt` | 0 | 0 |
| `ids_and_hashes.txt` | 0 | 0 |
| `numbers_and_contacts.txt` | 0 | 0 |
| `prose.txt` | 0 | 0 |
| `ruby_source.txt` | 0 | 0 |
| `safe_env.txt` | 0 | 0 |
| `source_code.txt` | 0 | 0 |
| `urls.txt` | 0 | 0 |
## How to reproduce
```bash
REDACTED_BENCHMARK_SEED=1792136801948421001 REDACTED_BENCHMARK_REPORT=docs/BENCHMARK.md cargo test detection_benchmark
```
## What this does not show
- Learned secrets are excluded. They match values from your own environment, which a fixed benchmark cannot hold.
- The hook envelope path (JSON in, JSON out) is tested in `tests/cli.rs` and `tests/golden.rs`, not here.
- No competitor numbers. Other tools were not run.
- The placements were chosen by the author. Real output can hold shapes not listed here.
- The floors run on fresh values on every `cargo test`. Counts on the expected-miss rows can move by a few between seeds.
- Contributions of failing samples are welcome: open an issue with the shape, never a real secret.