# Precision and false-positive methodology
This document describes how foxguard measures rule precision, how the built-in
rules are classified, and the known false-positive patterns we have seen and
tuned for. It is the honest, technical counterpart to the marketing-facing
coverage table in the [README](../README.md).
It is also the document [issue #9](https://github.com/0sec-labs/foxguard/issues/9)
asks for: rather than publishing a single global "precision number" without
context, we publish a per-rule tier classification, a labeled OSS precision
corpus, and an explicit list of the false-positive patterns we already know
about.
If you want the one-line version: foxguard rules fall into three tiers — tight
structural matches, intentionally-loud conservative sink rules, and
heuristic/regex-based rules — plus a fourth taint-based tier that trades recall
for precision. The labeled corpus is intentionally small and reviewed rather
than broad and unlabeled; use it as a regression baseline, not as a universal
claim about every repository.
## Section 1: Methodology
foxguard measures rule quality with four artifacts that all live in the repo:
1. **Positive fixtures** under `tests/fixtures/vulnerable_*.{py,js,rs,...}`
and `tests/fixtures/vulnerable.{py,js,go,rs,java,php,rb,cs,swift}`. These
are hand-crafted files with one vulnerable flow per function. Every handler
should fire exactly once for the rule it targets. The integration tests in
`tests/` assert that: if a positive fixture stops matching, CI fails. This
is our **recall floor**.
2. **Negative fixtures** under `tests/fixtures/safe*.{py,js,go}`,
`tests/fixtures/safe_py_taint.py`, `tests/fixtures/safe_js_taint.js`, and
`tests/fixtures/safe_py_aliases.py`. These contain patterns that *look*
dangerous but are not — for example, Django settings that read
`SECRET_KEY` from `os.environ`, SQL-ish strings with no interpolation, or
import aliases that rename safe functions. The integration tests assert
these produce **zero findings**. This is our **precision floor** for the
patterns we already know about.
Sitting between the one-function-per-case positive fixtures and the real-
world corpus scans is a **realistic test corpus** under
`tests/fixtures/realistic/`. Each file there is a small-but-complete
vulnerable application for one supported framework (Flask, Django,
FastAPI, a CLI tool, Express, Next.js, Hono) with idiomatic routing,
helper functions that exercise interprocedural propagation, and 2–3
explicitly-labelled `NEAR MISS` functions per file whose patterns the
engine might be tempted to flag but should not. The integration test
`tests/realistic_fixtures.rs` pins the exact total finding count and the
exact count per taint rule for every file. Any rule addition or engine
change that shifts these counts forces explicit acknowledgement — which
is how we catch end-to-end precision regressions that neither the
single-function fixtures nor the public corpus scan would notice.
3. **Labeled OSS precision corpus.** `benchmarks/precision/corpus.toml` pins
Express, Flask, Echo, and OWASP Juice Shop by full commit SHA. Every current
foxguard finding in those scan scopes has a row in
`benchmarks/precision/labels.jsonl` with one of `true_positive`,
`false_positive`, or `unsure`, plus a one-line review justification. Run:
```sh
./benchmarks/precision/run.sh --check
```
The command clones the corpus, scans it, joins findings to labels, and
writes `benchmarks/precision/results/summary.md` plus machine-readable
`summary.json`. The blessed snapshot in
`benchmarks/precision/expected.json` currently covers 48 findings:
16 true positives, 32 false positives, 0 unsure. Reviewed precision is
**33.3%** and reviewed noise is **66.7%** on this seed corpus. That number is
deliberately not softened: it shows the noisy rule families we need to fix,
especially broad path traversal, hardcoded-secret, and framework-helper
heuristics.
CI validates the corpus metadata on every PR, and the scheduled
`.github/workflows/precision-nightly.yml` run executes the full corpus,
uploads the dashboard artifact, and fails if reviewed precision drops or
reviewed noise rises beyond the thresholds in `corpus.toml`.
4. **Real-world corpus scans.** We have run foxguard against the latest HEAD of
Express, Flask, Gin, Laravel, Rails (actionpack), Spring Petclinic, and a
Next.js starter — 799 files, 369 findings, 0.86 seconds total. Numbers and
per-repo breakdown are in the blog post
[*I scanned Express, Flask, Rails, Gin, and Laravel for security
issues*](https://foxguard.dev/blog/scanning-top-frameworks). That scan is
the basis for the tier classifications in Section 2 — rules that produced
review-worthy findings on framework source code versus rules that stayed
silent are what informed the tight/conservative/heuristic split.
### Metrics we track
- **Recall on positive fixtures:** `(# findings on positive fixture) /
(# expected findings)`. For well-constructed fixtures this is 1.0, and CI
fails if it drops. This does **not** measure recall on real code — a hand-
crafted fixture is not a representative corpus.
- **Reviewed precision on the labeled corpus:** `TP / (TP + FP)` over labels in
`benchmarks/precision/labels.jsonl`, excluding `unsure` from the denominator.
The current seed-corpus snapshot is 33.3% reviewed precision and 66.7%
reviewed noise. This is a regression baseline, not a global product claim.
- **False-positive footprint on broader real code:** the count of findings our
engineers reviewed on the seven-framework corpus and classified as "not a
real issue." Section 3 lists every false-positive *pattern* we found and
fixed; the labeled precision corpus is now the durable place to track these
classifications over time.
- **Noise ratio:** findings in test/example/vendored directories versus
findings in production code. foxguard ships a built-in noise filter in
`src/engine/scanner.rs` that skips `vendor/`, `node_modules/`,
`__fixtures__/`, `__mocks__/`, `dist/`, `build/`, `.next/`, `coverage/`,
and `.cache/`, and also skips `.min.js`/`.min.css` files and any file whose
first line is longer than 1000 characters (a minified-bundle heuristic).
This is the single biggest FP-reduction lever and it runs by default.
### What this methodology is not
It is not a universal precision claim for all codebases. The labeled corpus is
a pinned, reviewed seed set designed to catch regressions and expose noisy rule
families. It will grow over time, but users should still treat the per-rule tier
classification and the known false-positive notes as the context for deciding
which rules to gate in their own repositories.
## Section 2: Rule classifications
Every built-in rule is placed in one of four tiers based on how its
implementation decides to fire. The tier is a property of the *rule design*,
not of a measured FP rate. The rule IDs below are the exact identifiers
emitted in foxguard's output and match the strings in `src/rules/*.rs`.
### Tier 1 — Tight (structural match on a specific shape)
These rules match a very specific AST or configuration shape where a positive
match is almost always a real issue. Expected FP rate is low.
**JavaScript / TypeScript**
- `js/jwt-none-algorithm` — flags `algorithm: 'none'` in a `jsonwebtoken`
options object. The shape is unambiguous.
- `js/jwt-hardcoded-secret`, `js/jwt-ignore-expiration`,
`js/jwt-decode-without-verify`, `js/jwt-verify-missing-algorithms` — all
structural matches on `jsonwebtoken` call shapes.
- `js/express-no-hardcoded-session-secret`,
`js/express-cookie-no-secure`, `js/express-cookie-no-httponly`,
`js/express-cookie-no-samesite`,
`js/express-session-saveuninitialized-true`,
`js/express-session-resave-true` — all match on Express
`session()`/`cookie()` option objects with a specific key/value.
- `js/no-cors-star` — `cors({ origin: '*' })` or equivalent.
**Python**
- `py/django-secret-key-hardcoded`, `py/flask-secret-key-hardcoded` —
assignment to `SECRET_KEY` with a string literal, excluding values that
come from `os.environ`/`os.getenv`.
- `py/flask-debug-mode`, `py/no-debug-true` — `app.run(debug=True)` and
`DEBUG = True` at module scope.
- `py/session-cookie-*`, `py/csrf-cookie-*`, `py/wtf-csrf-*`,
`py/django-allowed-hosts-wildcard`, `py/secure-ssl-redirect-disabled`,
`py/csrf-exempt` — all structural matches on Django/Flask-WTF config keys.
**Rust**
- `rs/tls-verify-disabled` — `danger_accept_invalid_certs(true)` in reqwest
or `danger().set_certificate_verifier(...)` in rustls.
- `rs/transmute-usage` — any call to `std::mem::transmute`. Rare enough in
real code to be worth flagging on sight.
**Go**
- `go/gin-no-trusted-proxies` — `SetTrustedProxies(nil)` on a Gin engine.
- `go/insecure-tls-skip-verify` — `InsecureSkipVerify: true` in a
`tls.Config` literal.
- `go/net-http-no-timeout` — `http.Server{...}` struct literal with no
`ReadTimeout`/`WriteTimeout` field.
**Java**
- `java/spring-csrf-disabled`, `java/spring-cors-permissive` — structural
matches on Spring Security builder calls.
- `java/no-xxe` — specific parser factory configurations.
**C#**
- `cs/no-xxe`, `cs/no-cors-star` — structural matches on
`XmlReaderSettings`/`UseCors` call shapes.
**PHP**
- `php/no-extract` — any call to `extract()` on a superglobal.
**Swift**
- `swift/no-tls-disabled`, `swift/no-insecure-transport`,
`swift/no-insecure-keychain` — configuration-value matches on
`URLSessionConfiguration`, `NSAppTransportSecurity`, and Keychain
attribute dictionaries.
Rationale: a match here requires the user to have written out the specific
dangerous shape. The regex-or-pattern surface is narrow enough that benign
code almost never looks like this.
### Tier 2 — Conservative (intentional high-recall sink rules)
These rules fire on *any* occurrence of a dangerous primitive, with no
attempt to decide whether the data is user-controlled. They are deliberately
loud. The point is recall: you will get a finding every time the primitive
is used, and it is the reviewer's job to decide whether that use is safe in
context. In exchange, you get zero false negatives on the primitive itself.
- `py/no-eval`, `py/no-pickle`, `py/no-yaml-load` — any call to `eval`,
`pickle.loads`, `yaml.load` (without `SafeLoader`). Alias-aware via
`ImportAliases`.
- `rs/unsafe-block` — every `unsafe { }` block.
- `rs/no-unwrap-in-lib` — every `.unwrap()` in a `lib.rs` target.
- `php/no-eval`, `php/no-unserialize`, `php/no-preg-eval`,
`php/no-file-inclusion` — any occurrence of the dangerous primitive.
- `rb/no-eval` — `eval` and `instance_eval` calls. `class_eval`/`module_eval`
are deliberately excluded (see Section 3).
- `java/no-unsafe-deserialization` — `ObjectInputStream.readObject` on any
stream.
- `cs/no-unsafe-deserialization` — `BinaryFormatter` / `NetDataContractSerializer`.
- `swift/no-eval-js` — `WKWebView.evaluateJavaScript`.
Rationale: if you use one of these primitives, you own the review. foxguard
will not pretend to know whether the input is sanitized. The rule doc string
says "avoid" or "review" rather than "this is definitely a bug."
### Tier 3 — Heuristic (pattern- or regex-based, context-dependent)
These rules match on patterns that *correlate* with bugs but need human
context to confirm. They are where most known false positives live.
- `js/no-sql-injection` — requires a SQL keyword followed by SQL structure
(`SELECT ... FROM`, `INSERT INTO`, `UPDATE ... SET`, `DELETE FROM`,
`DROP/ALTER/CREATE TABLE`, `EXEC`) in the literal, plus string
concatenation or template interpolation. See commit `13ea1ae` — earlier
versions matched `res.send('delete ' + name)` and were retightened.
- `py/no-sql-injection`, `go/no-sql-injection`, `rb/no-sql-injection`,
`php/no-sql-injection`, `java/no-sql-injection`, `cs/no-sql-injection`,
`rs/no-sql-injection`, `swift/no-sql-injection` — same family, same
limitation: without taint we cannot tell whether the interpolated value is
user-controlled.
- `js/no-hardcoded-secret`, `py/no-hardcoded-secret`, and the per-language
equivalents — regex on variable names (`password`, `secret`, `api_key`,
`token`, `auth`, `credential`, `private_key`) assigned to string literals
of length ≥ 4. Matches test fixtures with `password = "hunter2"` as
intended; will also match legitimate constants named `*_TOKEN_HEADER` or
`API_KEY_PARAM` that hold a parameter *name* rather than a value.
- `js/no-xss-innerhtml`, `js/no-document-write`, `js/no-unsafe-format-string`
— flag dangerous DOM sinks without deciding whether the argument is
tainted.
- `js/no-command-injection`, `py/no-command-injection`, and the per-language
equivalents — flag `exec`/`spawn`/`Runtime.exec`/`os.system`/`shell=True`
on any argument, not only tainted ones.
- `js/no-ssrf`, `py/no-ssrf`, `go/no-ssrf`, `rs/no-ssrf`, etc. — flag HTTP
client calls with any non-constant URL argument.
- `js/no-path-traversal`, `py/no-path-traversal`, etc. — flag filesystem
calls with path concatenation.
- `js/no-prototype-pollution`, `js/no-unsafe-regex` — AST patterns that
correlate with bugs but require reviewer judgement.
- `py/no-weak-crypto`, and `*/no-weak-crypto` family — flag `md5`, `sha1`,
`DES`, `RC4`. False positive on legitimate non-security uses (content
addressing, cache keys, Git-compatible hashing).
Rationale: these rules answer "is a dangerous primitive used with
non-constant input?" Without flow analysis we cannot answer the stronger
question "is that input user-controlled?" The taint rules below are the
precision-focused alternative.
### Tier 4 — Taint (flow-sensitive, precision-first)
These rules reuse the intraprocedural taint engine documented in
[docs/taint-tracking.md](./taint-tracking.md). They fire only when the engine
can prove a flow from a known source (e.g. `request.args.get`, `req.body`)
into a known sink (e.g. `pickle.loads`, `innerHTML`) within a single function
body, with no intervening sanitizer. They have **near-zero false positives
within their scope** but higher false negatives than their conservative
counterparts — anything interprocedural, cross-file, or involving a flow
shape the engine does not model will be missed.
Python source coverage spans Flask, Django, and FastAPI/Starlette request
attributes, plus common CLI-tool inputs (`sys.argv`, `sys.stdin`, `input()`,
`os.environ`, `os.getenv`). Handler parameters named `request` or `req` are
treated as implicit sources. See `python_taint_sources()` in
`src/rules/python_taint.rs` for the canonical list.
- `py/taint-pickle-deserialization`
- `py/taint-eval`
- `py/taint-command-injection`
- `py/taint-ssrf`
- `py/taint-yaml-load`
- `py/taint-sql-injection`
- `js/taint-xss-innerhtml`
- `go/taint-command-injection`
- `go/taint-sql-injection`
- `go/taint-ssrf`
Go source coverage spans net/http handler parameters (`r`, `req`,
`request`), Gin `*Context` methods (`c.Query`, `c.PostForm`, `c.Param`,
`c.GetHeader`, `c.FormValue`, ...), Echo `c.QueryParam`, Fiber
`c.Params` / `c.Query`, and generic `os.Getenv` / `os.Args`. The
engine propagates taint through method calls on tainted receivers,
`fmt.Sprintf` wrapping, `+` string concatenation, one-level selector
and index chains, and Go's native multi-return destructuring
(`a, b := f()`). Interprocedural return summaries are computed per
file and keyed by function / method simple name. See
`go_taint_sources()` in `src/rules/go_taint.rs` for the canonical
source list and `docs/taint-tracking.md` for the full scope.
The taint rules and the Tier-2 conservative rules are meant to coexist. If
you want loud-and-safe, keep the Tier-2 rules on. If you want
quieter-and-precise on a subset of classes, rely on the taint rules. Both
run by default.
## Section 3: Known false-positive patterns
These are patterns we have seen produce false positives and have explicitly
tuned for. Every item here is verifiable in the code or the git log.
- **SQL injection on plain English.** Earlier versions of
`js/no-sql-injection` flagged `res.send('delete ' + name)` because
`delete` is a SQL keyword. Fixed in commit
[`13ea1ae`](https://github.com/0sec-labs/foxguard/commit/13ea1ae) by
requiring a full SQL structure in the regex (`SELECT ... FROM`,
`INSERT INTO`, `UPDATE ... SET`, `DELETE FROM`, `DROP TABLE`, etc.) rather
than a bare keyword. See `src/rules/javascript.rs` around the
`NoSqlInjection` rule.
- **Minified JavaScript fixtures.** Bundled jQuery-style
`!function(e){...}(...)` triggered `js/no-command-injection` and
`js/no-eval` because the bundles genuinely contain `eval`/`Function`. We
skip `.min.js` files and files whose first line exceeds 1000 characters
(see `is_noise_path` and `is_minified` in `src/engine/scanner.rs`).
- **Rails metaprogramming.** `class_eval` and `module_eval` are standard
Ruby metaprogramming used by Rails, ActiveRecord, and every halfway-serious
Ruby gem. `rb/no-eval` only flags `eval` and `instance_eval`
(`src/rules/ruby.rs` around line 86).
- **Django `SECRET_KEY` from the environment.**
`py/django-secret-key-hardcoded` and `py/flask-secret-key-hardcoded` do
not fire when the RHS is a call to `os.environ.get`, `os.getenv`, or a
subscript of `os.environ`. The negative fixture
`tests/fixtures/safe.py` covers this.
- **Test/example/generated code.** foxguard's default noise filter skips
`vendor/`, `node_modules/`, `__fixtures__/`, `__mocks__/`, `dist/`,
`build/`, `.next/`, `coverage/`, `.cache/`, plus `.min.js`/`.min.css`
files and any file that looks minified by the first-line-length heuristic.
This is the single biggest source of noise in a default scan and it is on
by default. Users who want to scan these paths can pass them explicitly.
- **Import aliases.** `py/no-pickle` and friends are alias-aware
(`src/rules/python_aliases.rs`). `from pickle import loads as deserialize`
followed by `deserialize(x)` fires the rule; `from safe_module import loads`
followed by `loads(x)` does not.
## Section 4: How to report a false positive
If foxguard fires on code you believe is safe, open a GitHub issue with the
label **`false-positive`**. Include:
1. The rule ID exactly as emitted (e.g. `js/no-sql-injection`).
2. The smallest self-contained source snippet that reproduces the finding.
3. One or two sentences explaining why the finding is not a real issue —
what the reviewer would have checked in their head.
We treat false-positive reports as bugs against the rule, not against the
user. If the pattern is general enough to add to the negative-fixture suite,
we do that as part of the fix so it cannot regress silently.
## Section 5: How to suppress a finding
Two mechanisms, documented more fully in the main
[README](../README.md#suppressing-deliberate-findings):
- **Inline ignores** — the `foxguard: ignore` and `foxguard: ignore[rule-id]`
comment directives suppress a finding on the next non-empty, non-comment
code line (or on the current line when placed as a trailing comment).
Use these for one-off deliberate patterns. Inline ignores apply to code
scanning findings, not to `foxguard secrets`.
- **Repo-local baselines** — `scan.baseline` and `secrets.baseline` in
`.foxguard.yml` (see the [Configuration](../README.md#configuration)
section of the README) snapshot the existing set of findings so legacy
issues stop blocking adoption. New findings still fail the run; previously
accepted findings do not.
Prefer inline ignores for deliberate patterns (it keeps the justification
next to the code) and baselines for gradual rollout on legacy repositories.
---
*This document now includes the labeled precision corpus added for
[issue #445](https://github.com/0sec-labs/foxguard/issues/445).*