Skip to main content

Module scrub

Module scrub 

Source
Expand description

Credential scrubbing and safe truncation (plan §3, roadmap M5).

Every transcript line passes through scrub before it is written to disk or broadcast as a worker.message event, replacing common credential shapes with [REDACTED]. This is defense in depth, not a guarantee — the permission layer (§4.7) is the primary control.

The rule set is a fixed, OnceLock-compiled list of regexes plus one entropy-gated pass. There are deliberately no external dependencies (no secret-scanning crate): everything is regex + a hand-rolled Shannon entropy helper. The design goal is high recall on real credential shapes while keeping false positives low enough that ordinary prose, git SHAs, UUIDs, and placeholder tokens survive untouched.

§Rule ordering

Rules run in list order and the output of each feeds the next, so the most specific patterns must run first:

  1. PEM private-key blocks (multi-line) — removed whole before anything inside them can match a narrower rule.
  2. GCP service-account private_key JSON — the escaped PEM body that lives on a single JSON line.
  3. Vendor-specific fixed-prefix tokens (Anthropic, OpenAI incl. sk-proj-, Google AIza, Stripe, npm, GitHub, AWS, Slack, JWT). These have unmistakable shapes, so they run before any generic rule.
  4. Credential headersAuthorization: Bearer / Basic. Scheme word kept, credential redacted.
  5. Connection-string passwordsscheme://user:PASSWORD@host. Only the password segment is redacted; user and host stay for diagnosis.
  6. Generic assignment catch-allkey/secret/token/password = value. Runs late so a vendor rule gets first crack at the value. Implemented as an allowlist-aware closure pass (not a static replacement) so placeholder tokens, UUIDs, and git SHAs assigned to secret-ish names survive.
  7. Entropy-gated bare token — a high-entropy base64/hex blob that sits next to a broader secret-ish key name (access_token, client_secret, auth, …) not covered by rule 6. This is the only rule that reasons about the content of the value, and it is gated behind both a key-name context match and the allowlist plus a 4.0 bits/char entropy floor, so random-looking prose, git SHAs, and UUIDs are never touched.

Rules 1–5 are static OnceLock regex replacements; rules 6–7 are OnceLock-compiled regexes applied through closures so they can consult the allowlist and (for rule 7) entropy. Every pass is deterministic.

truncate_chars cuts long content on a char boundary so multibyte text can never panic the engine or produce invalid UTF-8.

Structs§

SecretFinding
A secret detector hit. Never carries the secret value itself.
SecretScan
Result of scanning and redacting a text payload.

Constants§

SECRET_ALLOWLIST_PATH
Tracked repository file containing one waived secret fingerprint per line.

Functions§

filter_allowed
format_findings
read_allowlist_text
scan_paths
Scan file contents about to be committed by the engine.
scan_text
Find secrets in text.
scan_text_at
Find secrets in text, using location only for diagnostics.
scan_unified_diff
Scan only added lines in a unified git diff.
scrub
scrub_and_truncate
scrub then truncate_chars — scrubbing happens first so truncation can never split a secret into an unrecognizable (and unredacted) prefix.
scrub_json_value
Redact every string leaf in a JSON value. Findings carry JSON-pointer-ish locations rooted at location.
scrub_with_findings
Scan and redact anything that looks like a credential.
truncate_chars
Truncate to at most max characters (not bytes), appending … [truncated] when anything was cut. Always cuts on a char boundary, so multibyte input can never split.