cerberust 0.1.1

Fast Rust guardrails for LLM input/output — composable scanners (PII, secrets, prompt-injection) and streaming middleware.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

# cerberust vs LLM Guard — benchmark results

Speed and detection parity of cerberust (Rust) against
[LLM Guard](https://github.com/protectai/llm-guard) (Python) on a shared
290-sample labeled corpus (100 PII positives, 96 secret positives, 30 prompt
injections, the rest clean / near-miss negatives). See [README.md](README.md)
for how to reproduce; both sides read the same `corpus/corpus.jsonl`.

These are **real measured numbers** from one machine (Apple Silicon, macOS).
Throughput is hardware- and load-dependent — the speedup ratios are the durable
signal, not the absolute samples/sec. The table is regenerated by
`compare.py` from the two result JSONs; the prose around it is hand-written.

<!-- BENCH-TABLE:BEGIN -->
| Scanner (vs LLM Guard) | cerberust speed | LLM Guard speed | speedup | cerberust P/R | LLM Guard P/R |
|---|---|---|---|---|---|
| PII (Anonymize) | 1.34M/s | 65/s | 20523× | 1.00 / 1.00 | 0.75 / 0.89 |
| Secrets (Secrets) | 1.31M/s | 1.4k/s | 940× | 1.00 / 1.00 | 1.00 / 0.45 |
| Regex (Regex) | 7.81M/s | 124.5k/s | 63× | 1.00 / 1.00 | 1.00 / 1.00 |
| BanSubstrings (BanSubstrings) | 1.16M/s | 135.5k/s | 9× | 1.00 / 0.27 | 1.00 / 0.27 |
| PromptInjection (PromptInjection) | 126/s | 96/s | 1× | 0.30 / 0.93 | 0.30 / 0.93 |
<!-- BENCH-TABLE:END -->

*Speed is samples/sec over the whole corpus. The regex/literal scanners (PII,
Secrets, Regex, BanSubstrings) are timed across repeated warm passes; the ML
PromptInjection scanner is timed once per sample — inference dominates. P/R is
precision/recall of each scanner's flag against the corpus ground-truth label.*

## What "the same comparison" means per row

- **PromptInjection — the clean apples-to-apples.** Both sides run the *same
  model* (`protectai/deberta-v3-base-prompt-injection-v2`) at the *same 0.5
  threshold*; the only difference is the runtime — cerberust's `ort` (ONNX
  Runtime, CPU) vs LLM Guard's `transformers` (PyTorch, MPS on this box).
  Detection is **byte-identical** (P=0.30 R=0.93, same 28 TP / 65 FP / 2 FN /
  195 TN). That is the headline parity result: cerberust's inference path
  reproduces LLM Guard's classification exactly. The low precision is a property
  of *this model at 0.5* on a corpus of short benign prompts, not of either
  implementation. (LLM Guard's own default threshold is 0.92, which trades recall
  for precision; we pinned both to 0.5 for a like-for-like comparison.)
- **Regex / BanSubstrings — identical patterns, identical detection.** Both run
  the same email regex / the same five trigger phrases, so P/R matches by
  construction; the comparison is pure matcher speed. BanSubstrings recall is
  0.27 on both because a five-phrase keyword list only covers a third of the
  injection set — that is the honest ceiling of a literal-substring baseline, and
  is exactly why the ML scanner exists.
- **PII / Secrets — different detectors, same job.** cerberust uses regex +
  checksums (Luhn, IPv4 range, entropy backstop); LLM Guard uses Presidio NER
  (PII) and detect-secrets (Secrets). Detection differs because the *approaches*
  differ, so these rows compare both speed and detection quality on structured
  PII / known-format secrets.

## Summary

cerberust wins decisively on the deterministic scanners and ties the ML scanner.

- **Speed:** 9×–20,000× faster on every regex/checksum/literal scanner. The PII
  and Secrets gaps are the largest because LLM Guard runs a spaCy NER pipeline
  (PII) and a multi-rule scanner (Secrets) in Python where cerberust runs compiled
  regex + checksums. The ML scanner is ~1.3× faster here even though LLM Guard had
  GPU (MPS) and cerberust ran ONNX on CPU.
- **Detection parity:** on this structured corpus cerberust matches or beats LLM
  Guard. It is perfect (P=R=1.0) on PII and Secrets; LLM Guard's Presidio NER
  misses/over-flags some structured PII (P=0.75 R=0.89) and its detect-secrets
  recall is 0.45 (it does not recognize the OpenAI/Stripe/labelled-`key=value`
  forms in the corpus). The ML PromptInjection scanner is **identical** to LLM
  Guard's — same model, same verdicts.
- **Honest gaps.** (1) The PII/Secrets detection edge reflects a corpus of
  *structured* entities, which regex+checksum handles natively; on free-text
  names/addresses (NER's strength) Presidio would lead until cerberust's planned
  `gline-rs` NER detector lands. (2) BanSubstrings recall (0.27) is low on both
  sides by design — it is a keyword baseline, not a detector. (3) The ML
  scanner's absolute throughput depends heavily on hardware; the 1.3× here is not
  a durable multiple the way the deterministic-scanner ratios are.

## Reproduce

```sh
# 1. corpus (no heavy deps)
python3 benchmarks/generate_corpus.py

# 2. cerberust side (Rust)
CERBERUST_MODEL_DIR="$HOME/.cache/cerberust/deberta-v3-base-prompt-injection-v2" \
  cargo bench --features prompt-injection --bench comparison

# 3. LLM Guard side (Python venv with llm-guard installed)
benchmarks/.venv/bin/python benchmarks/run_llm_guard.py

# 4. merge the fresh numbers into the table above
python3 benchmarks/compare.py
```