Expand description
The structured log sink, and the redaction that is not optional.
07-security.md’s threat table, on “Logs disclose repository secrets”:
“Structured allowlist logging with unconditional redaction of tokens,
headers, JIT blobs, and paths.” Its security gate is a secret-injection
log scan, and its release gate is that “the user access token and the
encoded JIT configuration are absent from logs, databases, snapshots, crash
reports, and CLI output”.
§Allowlist, not denylist — and why that is the whole design
A denylist redacts the fields somebody remembered to name. It is correct on
the day it is written and wrong on the day a later task adds a field, which
is the day it matters. So this sink inverts the default: a field whose
name is not in ALLOWED_FIELDS has its value replaced outright. A task
that adds runner_token to a log call gets [redacted] with no review, no
ceremony, and no leak. Adding a field to the allowlist is a deliberate edit
to this file that shows up in a diff.
The field name itself is kept. Knowing that an event carried a
runner_token field is useful, and the name is not the secret.
§Two layers, because one is not enough
The allowlist protects fields. It cannot protect the message body, which
has to be allowed or the logs say nothing — and a message body is exactly
where a secret ends up when somebody writes
info!("failed with {authorization}"). So every string that survives the
allowlist is then scrubbed by value shape (redact): GitHub token
prefixes, credential header names, long opaque runs such as an encoded JIT
configuration, and filesystem paths.
Scrubbing by shape over-redacts, deliberately. Some of that is worth knowing about before it surprises somebody:
- A slash-rooted word is treated as a filesystem path, and a URL path on
its own looks exactly like one. Log a full URL —
https://api.github.com/…— and it survives intact; log a bare/repos/owner/repoand it becomes[path]. - Any unbroken run of 40 or more base64, base64url, or hex characters is
treated as an opaque secret. There is exactly one carve-out: a value that
is precisely 64 lowercase hex characters is a SHA-256 digest, and renders
as a labelled 12-character prefix rather than disappearing. A digest is
not a secret, and
07-security.mdmakes checksum verification a security gate whose most useful diagnostic is expected-versus-actual. - A JSON Web Token is redacted whole. Its
.separators split it into runs the opaque-run rule is too short-sighted to catch, so it is recognised by shape instead: two or three base64url segments whose header beginseyJ. - A URL keeps its scheme, host and path and loses everything that
authenticates it: the
user:password@userinfo, the query string, and the fragment. The userinfo matters more than it looks — a token-authenticated git remote ishttps://x-access-token:ghu_…@github.com/owner/repo.git, so it is the shape a clone or fetch failure arrives in. The path stays diagnosable but is not exempt: each segment goes through the same shape rules on its own, because a token in…/raw/ghu_…/fis a token, and the alternative was the one place the belt never ran. A 40-character git object name is opaque enough to go with it. - A word ending in
:or=whose stem is a credential header name causes the next two words to be redacted, soAuthorization: Bearer ghu_…loses both the scheme and the token.
A word is cut on structural punctuation — ,, ;, {, }, [, ],
<, > and & — before any of that runs, and each fragment is then judged
the way a whole word is: unwrapped, judged on its core, and re-emitted with
its punctuation put back. Without that cut only the first key/value pair
in a compact structure is ever examined, and redaction becomes a function of
field order: {"encoded_jit_config":"…"} was caught and
{"runner_id":42,"encoded_jit_config":"…"} was not, while
serde_json::to_string is what decides which of the two an error body is.
Nesting, a form-encoded body, a ;-separated connection string and a plist
element are all that same defect in different punctuation. So is an array
element, one step further down — a fragment judged with its quote still
attached matches no shape rule at all, and an array element is the one
fragment that has no key of its own to give it away.
Because the cut is flat, a credential’s value does not have to be in the
same fragment as the key that names it — {"password":["hunter2"]},
{"password":{"v":"hunter2"}} and a plist’s
<key>password</key><string>hunter2</string> all put it one or more
fragments away — so a carry is threaded along the fragments to say that a
key is still waiting for its value, or that a quoted value was cut before
its closing quote. It steps over element names, because markup is not the
value a key named, and it stops at the punctuation that visibly closes the
value, because a redaction reported where no secret was is a false signal in
the one log a reader consults to find out whether anything leaked.
A URL is cut out of the text around it rather than being allowed to own the
rest of the word. Its scheme is the run of scheme characters immediately
before the ://, and it ends at the first character that cannot appear in a
URL — plus, ahead of its query string only, at a ; or an &, which are
structural characters everywhere else and are query syntax after the ?.
Everything on either side goes back through the rules. So
{"documentation_url":"https://…","token":"ghu_…"} — which is what a GitHub
REST error body looks like — keeps the URL and redacts the token, rather
than the URL swallowing the token, and
Server=https://vault.local/api;Password=… keeps the URL and redacts the
password rather than the URL swallowing the ; that separates them.
The text after a URL is iterated over, not recursed into. That is a
memory-safety property, not a style: recursing there made stack depth linear
in the length of the message, and ~86 KB of URL-carrying JSON exited
STATUS_STACK_OVERFLOW. A stack overflow is not catchable and takes the
process with it, so an attacker-influenceable error body could kill the
agent from inside its own log sink — and a sink that is not running redacts
nothing at all. For the same reason every search in split_url is bounded
by the URL’s own end: a pass that never returns is as effective a denial as
one that overflows.
A key is also trimmed of backslashes, which a value never is: Debug on a
String escapes the quotes inside it, so a body reached through
error!(reason = ?err) spells its keys \"password\". The trim_key
function documents why the same trim must not be applied to a value; it is
named rather than linked because it is private and this module’s
documentation is not.
Every one of those is a case where being wrong costs a slightly less readable log line, against a case where being wrong the other way costs a disclosed credential.
§What this does not do
It does not stop a caller printing to standard output, and it does not
redact a Debug derive somewhere else in the program. Those are held by
different controls: secrecy::SecretString for the two values that matter
(07-security.md’s credential inventory), and
crate::process::SpawnSpec::spawn_with_handoff for the command line.
Structs§
- Logging
Guard - Keeps the background log-writing thread alive.
- Redacting
Layer - A
tracinglayer that writes one redacted JSON object per event.
Enums§
- LogRole
- Whose diagnostics these are, and therefore which file they go in.
- Logging
Error - The sink could not be installed.
Constants§
- ALLOWED_
FIELDS - The field names this sink emits verbatim. Everything else is replaced with
REDACTION. - OPERATOR_
LOG_ STEM - The file stem
LogRole::Operatorwrites. - PATH_
REDACTION - What replaces something that was recognisably a filesystem path. Distinct
from
REDACTIONso a reader can tell “a path was here” from “a secret was here” without either being disclosed. - REDACTION
- What replaces a value this sink will not emit.
- SERVICE_
LOG_ STEM - The file stem
LogRole::Servicewrites, and whatservice statusreports as the daemon’s log.
Functions§
- install
- Installs the redacting sink as this process’s global subscriber, writing
daily-rotating files into
logs/. - is_
field_ allowed - Whether this sink will emit a field’s value rather than replacing it.
- redact
- Scrubs a string of anything that looks like a credential, an encoded JIT configuration, or a filesystem path.