forbidden-strings

Linear-time deny-list scanner for Git repos, built on the in-house forbidden-regex engine (package/rust-module/forbidden-regex). A native Rust binary with a sub-commit-budget startup, it scans working-tree files line by line against a deny list of literals and restricted-dialect regexes and reports each match as an opaque, redacted finding.

Rules split into a baseline embedded in the binary (data/builtin-rules.txt, activated by --builtin-rules), a committed shared appendix (forbidden-strings.append.txt), a per-repo gitignored appendix (forbidden-strings.append.local.txt), or a CI-only secret (FORBIDDEN_STRINGS_LIST). The matched substring, the surrounding line, and the rule pattern are never printed in failure output, so a rule body that would itself leak if committed (a customer name, an unreleased project codename, a pre-disclosure partner ID) can live as an appendix or CI secret without exposure on public CI logs.

What's different

• **Native binary startup. ** Rust with lto = true, codegen-units = 1, opt-level = 3, panic = "unwind", overflow-checks = true, strip = true. No Node startup and no WASM init. Explicit regex rules compile once per distinct content; later invocations reload compact exact-literal groups plus regex-only engine bytes and rebuild one Aho-Corasick matcher, which meets the measured sub-100 ms pre-commit budget.
• **Linear-time matching. ** The engine is derivative and product based with no backtracking, so no rule combination can exhibit catastrophic-backtracking behaviour. A set-level SIMD prefilter lets clean lines skip per-rule work.
• **Set-algebra rules. ** Intersection A & B and complement ~(A) are first-class in the dialect, so "match X but not Y" needs no lookaround. PCRE-family engines (gitleaks, trufflehog, secretlint, plain RE2) cannot do this; their workaround is per-rule allowlists, which scale badly.
• **Sensitive rules can live out-of-band. ** The committed baseline holds non-sensitive rules; the gitignored appendix and the CI-only FORBIDDEN_STRINGS_LIST secret hold sensitive rules. Failure output never prints the matched substring, the surrounding line, or the rule pattern, so a rule body itself can be a secret.

When to pick something else

forbidden-strings deliberately omits features other scanners ship as core capabilities:

• CEL-based post-match filtering (entropy thresholds, BPE token efficiency, git-author predicates, file-path globs, string allowlists). No equivalent here.
• Async HTTP validation. No way to call a provider API to confirm a detected secret is live; staleness review is on you.
• Git history scanning. The walker enumerates working-tree files only. No equivalent of gitleaks git that scans every diff in every commit.
• SARIF / JSON / CSV output. Hits go to stderr as plain text; no machine-readable format for GitHub code-scanning upload or CI dashboards.
• Per-rule path scoping. Every rule runs against every non-skipped file; the scanner cannot apply rule X only to YAML files.
• Per-rule allowlists. No way to say "rule X but skip when it matches in path Y".
• **No streaming or stdin input. ** Files only; the walker enumerates from disk.

If you need any of those, betterleaks or gitleaks is the right tool.

Prerequisites

• Rust toolchain. Install via mise: mise install rust.
• mise itself, since build commands are mise run tasks.

Build

mise run //package/cli/forbidden-strings:build

The release binary lands at package/cli/forbidden-strings/target/release/forbidden-strings. Root cli-git.config.ts gives that path to the bundled security/forbidden-strings policy; nothing needs to be on $PATH.

Setup

The scanner needs exactly one rules file at scan time. How you produce it is up to you.

Without file-enforcer (most consumers)

Put rules in a file named forbidden-strings.local.txt at the repo root (named tail-format sections, or legacy one-rule-per-line; see "Rule file format"), or pass --rules <PATH> / set FORBIDDEN_STRINGS_RULES=<PATH> to point at any other path. That is the whole setup. For a zero-file start, pass --builtin-rules to scan with the embedded betterleaks-ported baseline (see "Built-in baseline" below). Add the file to .gitignore if the rules themselves are sensitive; otherwise commit it. The "Rule file format" section describes the line syntax. In CI, materialise the file from a secret (see "GitHub Actions" below) so the rule bodies never enter version control.

With file-enforcer (this monorepo's workflow)

Inside the Monochromatic monorepo, no rules file exists at the repository root (see doc/decision/gitignore-negations.md). The pieces:

• The betterleaks baseline ships inside the scanner binary (data/builtin-rules.txt, regenerated by the two-stage chain behind mise run //package/cli/forbidden-strings:generate:rules: the TS porter src/mise.port-betterleaks.ts, then the dialectport bin in package/rust-module/forbidden-regex.bench/src/bin/); repo invocations activate it with --builtin-rules (the cli-git policy sets builtinRules: true).
• forbidden-strings.append.txt is the committed shared appendix of non-sensitive repo-wide rules.
• forbidden-strings.append.local.txt is the per-repo additions. Gitignored, free-form, edited by hand. Place sensitive literals (codenames, customer names, partner IDs) here.
• .cache/forbidden-strings.rules.txt is the authoritative runtime text consumed by the scanner. File-enforcer concatenates the two appendixes into the gitignored .cache/ scratch dir. The generated root mise.toml [env] points FORBIDDEN_STRINGS_RULES at it (absolute via {{config_root}}). Do not edit directly.
• A compiled runtime artifact lives under the current user's native cache root, partitioned by scanner version, platform, and SHA-256 of the exact runtime text. It stores exact-literal groups directly and precompiled engine bytes only for explicit regex rules. File-enforcer invokes forbidden-strings compile-rules after generating the text when the release scanner exists. A later scan repairs a missing or rejected artifact automatically.

Run mise run file-enforcer after editing either appendix to regenerate the runtime text and eagerly compile its cache artifact. The generator is generateForbiddenStringsRules in file-enforcer.config.ts. During fresh setup before the release scanner exists, file-enforcer reports that eager compilation was skipped; the first later scan compiles and publishes the artifact. A stale pre-0.4.0 release binary exists but does not understand compile-rules, so file-enforcer fails instead of silently claiming eager compilation; rebuild the scanner before rerunning file-enforcer. If you fork this scanner into a project that does not use file-enforcer, drop the appendix split and follow the single-file workflow above.

Usage

# scan a specific file list (uses ./forbidden-strings.local.txt by default)
forbidden-strings path/to/file other/file

# scan every working-tree file (.gitignore respected; .git/.jj skipped)
forbidden-strings --all

The rules path is resolved in this order: --rules <PATH> flag (highest), then FORBIDDEN_STRINGS_RULES environment variable, then ./forbidden-strings.local.txt in the current working directory.

Every runtime-rules scan uses read-write caching by default with no opt-out. FORBIDDEN_STRINGS_CACHE_DIR may set an absolute cache root. Without the override, the scanner uses $XDG_CACHE_HOME or $HOME/.cache on XDG-oriented Unix, $HOME/Library/Caches on macOS, and %LOCALAPPDATA% on Windows.

# explicit path
forbidden-strings --rules ./other-rules.txt --all

# via env var (CI-friendly: materialize from a secret, then run)
FORBIDDEN_STRINGS_RULES=./materialized.txt forbidden-strings --all

# eagerly compile one rules file without scanning
forbidden-strings compile-rules --rules ./other-rules.txt

# print version and exit
forbidden-strings --version    # or -V

compile-rules requires an explicit text path, derives the user-cache destination, and prints nothing on success. It reuses a valid artifact and otherwise compiles, rechecks the source snapshot, and atomically publishes owner-only cache data.

Cache misses and rejected artifacts fall back to authoritative text and emit one compact JSON warning on stderr before findings. A failed cache write emits another warning but does not invalidate the already compiled scan. Warnings contain only closed reason and recovery tokens, never source paths, cache paths, digests, rule text, or operating-system error text.

Built-in baseline (--builtin-rules)

The binary embeds the betterleaks-ported baseline ruleset. The text form (data/builtin-rules.txt) is exported as the library constant forbidden_strings::BUILTIN_RULES; the scan path loads the baseline from a serialized RegexSet precompiled at build time (compiling the full baseline at each startup is not viable, so the cost is paid once during the build). It is pure opt-in: without the flag the scanner never reads it, so existing invocations behave exactly as before the flag existed.

With --builtin-rules:

• The baseline is appended after the resolved rules file. Baseline findings render as rule=<name> (the upstream betterleaks id), so they identify themselves regardless of position. Unnamed legacy rules fall back to numeric ids that do not shift: your rules keep ids 0..user_len and the baseline takes user_len... A runtime rule whose section name collides with a baseline name fails the load closed (the finding token must stay unambiguous).
• When no rules file resolves at all (no --rules, no env var, and no ./forbidden-strings.local.txt in cwd), the baseline alone is the ruleset; passing the flag is itself the configuration.
• An explicitly named missing file (--rules <path> or the env var pointing at a path that does not exist) still exits 2: silently scanning without your rules would be a false-clean result.

# zero-file quick start: scan the tree with the embedded baseline only
forbidden-strings --builtin-rules --all

# your rules plus the baseline
forbidden-strings --builtin-rules --rules ./rules.txt --all

--all and positional files are mutually exclusive in practice: if both are passed, the walker output silently overwrites the positional list. Use one or the other.

Rule file format

Two formats, autodetected per file on the first significant line; one file never mixes them.

Tail format (named sections)

A rule file is what tail --verbose -n +1 over per-rule files would produce: a ==> name <== header line opens a section whose body is one rule. The name (first character [a-z0-9], then any of [a-z0-9.-], unique within the loaded file) is the rule's identity: findings render it as rule=<name>.

• A body with exactly one significant line (non-blank, not #-leading) is classified by the two line shapes below, exactly as in the legacy format.
• A body with more than one significant line is one verbatim regex pattern handed to the always-verbose engine as-is (blank lines and first-column # comments inside it are the engine's own to consume).
• Any line whose trimmed form starts with ==> without being exactly a strict header fails the load closed with its line number (case typos, indentation, missing-space arrows, and genuine arrow content alike); write genuine ==> content in a regex body with the [=]=> reshape.
• Empty sections, duplicate names, and significant content before the first header are fail-closed load errors, each reported by line number only.

Legacy format (one rule per line)

A file whose first significant line is not a strict header parses line-by-line, byte-for-byte as before; such rules carry no name and findings fall back to rule=N numeric ids. Two line shapes (shared with single-line tail sections):

• A bare line is a case-sensitive literal. It is escaped into the engine's verbose dialect and matched as a substring. A literal under eight bytes is word-boundary gated at each end whose neighbouring byte is an ASCII word byte, so a short code like ACR matches the whole token ACR but no longer fires glued inside ACRYLIC; eight bytes and longer match anywhere, and over-matching in that direction is a ratified preference. If a short literal must match glued substrings too, write it as a regex (/ACR/).
• A line of the shape /PATTERN/FLAGS is a regex in the forbidden-regex dialect. The first / and the last / delimit the pattern. FLAGS is a trailing run of ASCII-lowercase letters; if the trailing run is not all-lowercase, the whole line is treated as a literal instead (so /foo/I is a literal scan for the seven bytes /foo/I, not a case-insensitive regex).

Empty and whitespace-only lines are ignored. A line whose first non-whitespace byte is # is a comment. One leading UTF-8 BOM is stripped from the source. An empty source (no non-blank, non-comment line) is a rule-file error.

Flags policy

The engine is always in multiline and verbose mode, so the only accepted flags are the ones those two modes already imply:

• m (multiline) and x (verbose) are accepted as no-ops and dropped.
• **Any other flag letter is a hard, fail-closed load error. ** Silently dropping an i or an s would change match semantics (case folding, dot-matches-newline), so the loader rejects the whole ruleset rather than weaken a rule. Need one of those locally? Restructure the pattern (for case-insensitivity, spell the alternatives: [Aa][Bb][Cc]).

Supported constructs

The dialect is a deliberately restricted subset (see package/rust-module/forbidden-regex/README.md for the full engine spec):

• Literals and the escapes \t, \b (word boundary), backslash-escaped metacharacters, and backslash-escaped whitespace.
• Character classes: [abc], [a-z], [a-zA-Z], negated [^...], and the shorthands \d \w \s \D \W \S (usable inside classes too).
• . matches any byte except a newline.
• Grouping and alternation: (?:a|b). Groups are non-capturing only.
• Bounded repetition: a?, a{3}, a{3,6}.
• Anchors: ^, $, \b. The word set is ASCII [A-Za-z0-9_]; ^ and $ anchor at line boundaries.
• Set algebra: intersection &, complement ~(...).

Matching is an unanchored search over a single line's raw bytes: a pattern matches if it matches any substring. Because verbose mode is always on, unescaped whitespace outside character classes is ignored, so a rule may be written across several physical characters for readability; to match a literal space use \ , \t, or [ ].

Set-algebra operators

Two top-level set operators that pure-PCRE engines lack:

• A & B (intersection): matches strings matched by both A and B.
• ~(A) (complement): matches strings that do NOT match A.

Operators & and | take single-atom operands: a literal, a class, ., an anchor, a (?:...) group, or a ~(...). A concatenation or a quantified atom must be wrapped in (?:...) to be an operand, so there is no operator precedence to remember. ~(...) is always parenthesized. A pattern that can match the empty string is rejected (unanchored, it would match everything), so ~(Y) alone is rejected while (?:X) & ~(Y) with a concrete X compiles. Example: ban any five-digit key except the all-zeros placeholder:

/(?:key_[0-9]{5}) & ~(key_00000)/

Rejected at compile time (fail-closed)

Anything outside the supported set is a hard compile error naming the offending rule's opaque index, and the whole load fails closed (a bad ruleset never degrades to a partial scan): *, +, unbounded {n,}, \xNN byte escapes, capturing (, lookaround and inline-flag groups ((? not followed by :), backreferences, unknown escapes, unbalanced brackets, stacked quantifiers, {n,m} with n greater than m, repetition whose expansion exceeds the engine's cap, and any pattern that can match the empty string.

Load errors are redacted: a compile error carries only the opaque rule index (0-based position in the compiled set) and the engine's own static reason; a tail-format structural error (near-header, duplicate name, empty section, pre-header content) carries only a source line number; rule text appears in neither. The redacted error type is LoadError in src/rule/frx/error.rs.

Output

For each violation:

PATH:LINE rule=<token>

• LINE is the 1-based line number.
• The rule token is the rule's section name for tail-format rules (and the betterleaks id for baseline rules), or the 0-based numeric engine id for unnamed legacy rules. Unnamed runtime rules take ids 0..user_len; under --builtin-rules the baseline's unnamed ids are offset past them. The finding is columnless: the engine reports per-line rule indices, not spans, so no COL_START..COL_END segment appears.
• One finding is emitted per (line, rule) pair.
• **The matched substring, the line content, and the rule pattern are never printed. ** Only the path, line number, and rule identity token appear, so a failing CI log never becomes a leak surface. Name sensitive rules with deliberately opaque section names (the name is the only rule-derived text that surfaces); a numeric legacy index is looked up against the local rule file.

Two synthetic findings keep the scan fail-closed:

• **Read errors. ** A file that cannot be opened (broken symlink, permission denied, deleted during scan) produces PATH: read error: <reason> on stderr and counts toward the exit-1 total. A secret-scanning gate must not pass silently on a file it could not inspect.
• **Engine errors. ** If the matcher panics on a file, the catch_unwind boundary in scan_one_set (src/frx_scan.rs) catches it and emits PATH: engine error, again counting toward exit 1 rather than aborting or exiting clean.

Hits go to stderr, not stdout; redirecting 2>/dev/null silently loses the report. Within a file, findings are emitted in set order (runtime rules before the baseline), then by line; across files, ordering is rayon-scheduler-determined, stable on a given input but not alphabetic. Callers that need deterministic cross-file reports should pipe the output into sort.

Exit codes:

• 0: no violations.
• 1: one or more violations (real hits, read errors, or engine errors).
• 2: usage error or rule-file error.

Security model

The redaction guarantee is what lets a rule body itself be a secret. Two boundaries carry it:

• **Load path. ** Rule compilation reports only LoadError (src/rule/frx/error.rs), whose every variant is an opaque index plus the engine's static reason; no pattern bytes reach a diagnostic. The compiler builds through RegexSet::new / RegexSet::from_bytes, neither of which logs the pattern.
• **Scan path. ** Findings are formatted as PATH:LINE rule=N in src/frx_scan.rs; the matched bytes and the line content are never included. The fail-closed catch_unwind boundary emits only PATH: engine error.

Keep sensitive rule bodies out of tracked files: use the gitignored forbidden-strings.append.local.txt or the CI-only FORBIDDEN_STRINGS_LIST secret, never the committed baseline or appendix. Treat compiled artifacts as sensitive as their source: the runtime envelope retains exact bare-literal bytes and is confidential policy data, not a redaction. On Unix, file-enforcer enforces mode 0600 on the local appendix and generated runtime text; the scanner enforces private application directories and mode 0600 on artifacts.

In CI, pipe secrets through printenv rather than interpolating them into a workflow command; shell expansion can leak values even when log masking is enabled. Hosted GitHub runners use their native account cache for the derived artifact. A self-hosted or shared runner should set FORBIDDEN_STRINGS_CACHE_DIR to job-scoped storage when its account cache survives jobs.

Integration

Local cli-git policy

Root cli-git.config.ts enables security/forbidden-strings at error severity. The PATH-shadowed cli-git wrapper evaluates selected would-be-committed bytes before commit, landed commit bytes before automatic push, and Git-native outgoing ranges before manual push. Native --no-verify skips Git hooks but does not skip this policy.

Run an explicit read-only check through the built shim with:

git cli-git check --policy security/forbidden-strings --all

The policy invokes the repository-built scanner directly. Scanner infrastructure failures remain distinct exit-2 engine failures; findings exit 1.

GitHub Actions

.github/workflows/forbidden-strings.yml remains independent of cli-git trust and local wrapper state. It downloads the release matching the scanner crate version, verifies the archive's GitHub build-provenance attestation, materializes the committed baseline plus shared appendix and optional repository secret, then invokes the scanner binary directly. Pull-request and merge-queue jobs scan changed files relative to origin/main; pushes to main scan files changed across the push event's before and after commits. The same precedence applies locally and in CI: --rules > FORBIDDEN_STRINGS_RULES > ./forbidden-strings.local.txt.

Walker behaviour

• **--all semantics. ** Walks the working tree via ignore::WalkBuilder in src/walk.rs: .hidden(false) (dotfiles like .github/, .npmrc ARE scanned), .ignore(false) (the .ignore file is NOT consulted; .gitignore stays enabled). Files force-added past .gitignore (git add -f) are recovered via an in-process gix-index read of .git/index; no git subprocess.

• **.git/ and .jj/ skipped. ** Internal VCS state is never scanned.

• **Symlinks NOT followed. ** WalkBuilder's default follow_links is false; symlinked directories are not descended, symlinked files surface as a read-error synthetic hit on a broken target.

• **Non-UTF-8 paths silently dropped. ** Index entries that are not valid UTF-8 are excluded from the walk.

• **Binary-file 8 KiB tail cap. ** Files whose first 8 KiB contains a NUL byte are scanned only in the first 8 KiB. The leading window always runs, so secrets there fire; the tail past 8 KiB is skipped. Constant BIN_PROBE_SIZE and read_with_binary_check in src/lib.rs.

• **Self-skip set. ** During --all, canonical paths are auto-skipped so rule bodies do not self-match: the materialised rules file (whatever --rules / env var / default resolves to), plus three generated-source paths:

  • package/cli/forbidden-strings/data/betterleaks-default-config.toml
  • package/cli/forbidden-strings/data/builtin-rules.txt
  • package/cli/forbidden-strings/src/port-betterleaks-relaxations.ts

  Skip is path-anchored via std::fs::canonicalize, not basename-anchored, so an unrelated file named forbidden-strings.local.txt in a subdirectory is still scanned. Paths that fail to canonicalize from the current cwd are silently dropped from the set. Explicit positional arguments bypass the --all skip; the scanner's own forbidden-strings.*.txt config files at cwd are skipped in both modes (is_config_file_at_cwd in src/lib.rs).

Performance

The scanner is a native Rust binary tuned for a sub-commit-budget startup and linear-time matching; the release profile (Cargo.toml) sets lto, codegen-units = 1, opt-level = 3, panic = "unwind", overflow-checks = true, and strip. panic = "unwind" and overflow-checks = true are load-bearing for the fail-closed catch_unwind boundary, not speed: the forbidden-regex engine documents that it expects the caller's unwind boundary, and a wrapped overflow would otherwise let a corrupt rule fail open.

Full bench methodology and per-version regression history are in PERF.md. Runtime cache verification measures first compilation separately from repeated artifact loads. A valid artifact must preserve finding identity and matching behavior before its timing is accepted. The cache directory has no automatic retention policy; deleting its forbidden-strings/ child is always safe because authoritative text regenerates it.

Fuzzing

Coverage-guided fuzzing lives in its own package, package/fuzz/forbidden-strings, so a scoped nightly toolchain does not force this published crate onto nightly. The scanner exposes a curated internal surface (fuzz_api, behind the fuzzing Cargo feature) for the targets to drive. The teardown that removed the old engine also retired the gate, shard, and routing targets that fuzzed it; the surviving targets are retargeted onto the forbidden-regex load and scan path. See that package's README for prerequisites, mise commands, the bounded-container wrapper, and crash-reproduction guidance.

Architecture

• **Two-form loader. ** src/rule/frx owns the rule-file format and preserves whether each rule was written as a bare literal or an explicit regex. Bare literals still escape into the verbose dialect for the public engine compiler; runtime scanning instead retains their exact bytes. /PATTERN/FLAGS lines and multiline tail sections remain restricted regex rules, validated under their original global rule ids.
• **Hybrid runtime matcher. ** src/runtime_matcher.rs de-duplicates exact literals into one overlapping Aho-Corasick matcher and compiles only explicit regex rules into RegexSet. Both subset-local outputs map back to original ids, sort, and de-duplicate before finding attribution.
• **Content-addressed runtime cache. ** src/runtime_cache/ hashes exact authoritative text, selects a scanner-version and platform partition, validates scanner-owned envelope metadata, rebuilds the literal matcher from compact groups, and decodes optional regex-only engine bytes. Scan-time repair and compile-rules share the same atomic publisher.
• **Line-based batch scan. ** src/frx_scan.rs splits each file's bytes into lines once and hands the buffer plus line-start offsets through a common matcher interface. Runtime sets merge Aho-Corasick and regex-subset ids; the built-in baseline retains RegexSet::line_matches. Each set runs under a catch_unwind boundary so a matcher fault fails closed as a synthetic finding.
• **Build-time baseline precompilation. ** build.rs compiles data/builtin-rules.txt through the engine once at build time and serializes it (to_bytes); lib.rs embeds the blob with include_bytes! and the loader rebuilds it via the validating from_bytes, never recompiling. This trusted embedded path remains separate from the mutable runtime cache envelope. Cargo compiles build scripts' dependencies unoptimized by default, so Cargo.toml sets build-override opt-level = 3 for the release and dev profiles, and the engine builds the baseline's rules on one worker thread per core (see the Cargo build-override write-up).
• **Concurrent load and walk. ** Rule loading and --all file walking run concurrently via rayon::join (they share no state); files then fan out across the rayon thread pool for the parallel scan.
• **ignore crate walker + in-process gix-index union. ** --all uses ignore::WalkBuilder (honouring .gitignore, .git/info/exclude, and global excludes) and unions the result with an in-process gix_index::File read of .git/index so git add -f files are still discovered. See src/walk.rs.
• **Bundled data/betterleaks-default-config.toml. ** Upstream-vendored provenance for the betterleaks port; the embedded baseline is derived from it, and port-betterleaks-relaxations.ts records the lossy translations applied during the port.