Module secret_gate 

Write-time secret detection gate.

Scans caller-supplied content strings before any storage write. A match causes a hard RuntimeError::SecretDetected that names the detector and carries a masked excerpt — it never echoes the full candidate back.

Scope: credentials only — API keys, tokens, private keys, passwords, and connection strings with embedded credentials. General PII such as email addresses, phone numbers, and company names is intentionally NOT blocked; those are normal knowledge-graph content.

Detection is layered, cheap-first:

1. Known-prefix / known-shape patterns — AWS AKIA/ASIA, GitHub tokens, OpenAI sk-proj-, Anthropic sk-ant-, Stripe live keys, Fly.io tokens, Vercel secrets, Slack xox*, JWT triples, PEM private-key headers, Age secret keys, URL userinfo (scheme://user:pass@). Bare sk- is also checked but only when NOT followed by a known safe word boundary (e.g. sk-learn, sk-image).
2. High-entropy token heuristic — base64/hex/base64url runs ≥ 24 chars near a trigger word (key, secret, password, credential, bearer, auth, apikey, api_key, access_key, private_key). The word token alone is NOT a trigger to avoid blocking tokenizer_*, token_count, etc.

Allowlist (false-positive suppression) — all of the following are prose-context exemptions, not unconditional passes: a credential trigger word in the surrounding window always dominates. A UUID or a sha-prefixed content hash sitting directly beside “api_key”/“secret”/“auth” is exactly as ambiguous as any other high-entropy candidate and falls through to explicit detection instead of being silently allowed.

• Pure hex strings (sha256, git SHA) — passed when not near a trigger.

• UUID canonical form (xxxxxxxx-xxxx-…) — passed when not near a trigger.

• Base64/base64url content hashes with an explicit sha<N>- prefix (SRI hashes, npm lockfile integrity) — passed when not near a trigger and not preceded by a known-vendor prefix. Bare base64 tokens without the sha<N>- prefix are NOT passed.

• Strings that are entirely ASCII punctuation/whitespace (e.g. code) — not subject to the entropy heuristic, only the literal-prefix checks apply.

• Non-ASCII characters (CJK prose, accented text, emoji) act as token delimiters for the entropy heuristic: only maximal ASCII runs are entropy-checked. Real base64/hex/base64url credentials are ASCII, and shannon_entropy runs over UTF-8 bytes — multibyte codepoints inflate the byte-wise entropy and false-positive on natural-language non-Latin content. Treating non-ASCII as a delimiter (rather than skipping any whitespace token that merely contains it) keeps CJK prose unflagged while still catching an ASCII credential glued to CJK text/punctuation/fullwidth whitespace. The literal-prefix checks (Layer 1) treat any non-ASCII-alphanumeric char (CJK, accented text, emoji) as a token boundary, so a known-prefix secret is caught whether the adjacent non-ASCII sits before the prefix (数据AKIA…) or after it (AKIA…数据).

• Structured identifiers: a token is only considered for this exemption when it contains at least one of /, -, _, or . (the gate); it is then decomposed into maximal alphanumeric runs by splitting on every non-alphanumeric character (not just the four gating separators — any other ASCII punctuation glued into the same whitespace token, e.g. a stray : or ,, also acts as a run boundary). A token exempts when it decomposes into two or more such runs and every run is letters-then-digits or pure digits, at most 24 chars long, with a low case-transition density. This covers content like fable-ops/ADR-DRAFT-adr079.md or local workspace artifact, which is otherwise indistinguishable from a high-entropy secret once glued into one whitespace token. Random base64/base62 secrets do not decompose this way: their case and digit placement is effectively uniform rather than word-shaped, so a hyphenated or underscored secret still fails this check and remains subject to the entropy heuristic below.

  This exemption applies ONLY outside an explicit credential trigger context. Signals that measure Shannon entropy over an attacker-chosen run boundary (e.g. requiring a trailing file extension, or an average per-run letter entropy below a threshold) are not sound near a trigger word: an attacker who controls where a credential’s separators fall can always choose run lengths whose entropy reads no higher than an ordinary short English path segment, since the measure only sees a character-frequency histogram, never word semantics. So near a trigger word, a structured-identifier-shaped token gets no exemption at all and falls through to the entropy heuristic like any other token. This is an accepted false-positive tradeoff on a small number of genuine paths/doc-slugs that happen to sit near a trigger word AND read above the entropy threshold on their own — see accepted_false_positive_adr_draft_path_near_trigger and its siblings for the specific repro cases this blocks, and the call site in check_entropy_heuristic.

Trigger-word matching only fires on genuine mentions, not substring collisions: bare trigger words (key, secret, password, passwd, credential, bearer, auth, apikey) are matched at a word boundary (contains_bounded_word), so auth does not fire inside authorized or authentication, nor key inside monkey/keyword. The three compound entries that already embed an underscore (api_key, access_key, private_key) are matched as a plain substring instead, since the underscore already disambiguates them and word-boundary matching would only weaken them (e.g. secret_access_key no longer needs the bare secret/key matches once access_key fires on its own).

A structured-identifier-shaped token sitting near a genuinely standalone trigger word (e.g. auth work saved at .../repo-audit.md, where auth is an actual topical mention rather than a substring collision) is an accepted false positive: no window-narrowing or exemption-widening scheme survives the adversarial regression corpus without also reopening a real bypass, because the caller (or an attacker) fully controls the prose between a trigger word and a payload: narrowing TRIGGER_WINDOW or reinstating the structured-identifier exemption near “bare” trigger mentions both fail the same known bypass strings that motivated closing them.

The caller-visible block message (SecretMatch’s Display impl) also carries actionable guidance (block_guidance) to split or reword the flagged token.

The word-boundary rule above treats underscore as a BOUNDARY for bare TRIGGER_WORDS (contains_bounded_word): deliberately different from has_standalone_token’s rule for the word token, which treats underscore as a continuation so tokenizer/next_token/token_count stay exempt. Treating underscore as a boundary for the bare set is what lets common underscore-joined credential-config compounds keep firing: SECRET_KEY=... (Django/Flask-style config), auth_token=..., session_secret_..., signing_key=... all match on the secret/key/ auth half even though none of them is in COMPOUND_TRIGGER_WORDS. This is implemented by parameterizing the boundary rule (contains_word’s underscore_is_word_char argument) rather than sharing one rule between the two callers.

A production-corpus replay harness (corpus_replay, #[ignore]d, run via KHIVE_REPLAY_DB=<path> cargo test ... -- --ignored --nocapture) measures the detector’s block rate against real note/entity content; see the harness’s own output for current numbers rather than a point-in-time count here, which would drift as the corpus changes.

Structs

SecretMatch

Returned when a write would store credential-looking content.

Functions

check

Hard-block content from being written.

check_json

Recursively scan a JSON value for credential-shaped strings.

check_tags

Scan a string-tagged slice (entity/note tags).

mask_secrets

Redact every detected secret span in text, replacing each with ***MASKED***.