acme-proxy 0.2.0

# CLAUDE.md — `src/`

Design and layout for the crate itself. The repository-wide file (`../CLAUDE.md`) holds the feature overview, the configuration keys, CI, the migration rules and the crate-wide conventions — including the nine logging rules every call site in this tree must follow. `tests/CLAUDE.md` holds the harness and the integration suites.

## Architecture

Request flow for a signed ACME POST (e.g. `newAccount`):

1. **`add_nonce_middleware`** (`src/middlewares/nonce.rs`) wraps every route of a profile but mints only where RFC 8555 asks (`mints_nonce`): **every response to a POST** (§6.5, errors included) and **`newNonce`** in all three of its forms (§6.3/§7.2). Not on `GET /directory`, `/crl`, `/renewalInfo/{id}` or the fallbacks — each of those used to cost a committed database write for a nonce nobody asked for, on the traffic that dominates a real deployment. The layer stays **outside** the filter layer, so a POST a filter refuses still carries one.
2. **`AcmeRequest<T>` extractor** (`src/extractors/acme.rs`) is the core of the codebase. Its shared `verify_jws` core **checks `Content-Type: application/jose+json`** (§6.2 — a mismatch is `415`, before the body is read, so a rewritten header never burns a nonce), parses the flattened JWS, base64url-decodes the protected header, **rejects any `crit`** (RFC 7515 §4.1.11: this server implements no critical extension, so every value is unrecognized), **verifies the signature with `ring`**, **checks the JWS `url` against the route actually reached** (§6.4) and **consumes the nonce** (§6.5). Hoisting these here makes them structural: a new signed route can't forget them. The account's `last_seen_*` stamp is here for the same reason — the one place every `kid`-authenticated request funnels through — and **after** the nonce check, so a replayed request never moves the mark; the reverse lookup runs only once `Account::needs_touch` says the write is worth doing. Three extractors build on it: `AcmeRequest<T>` (decode + deserialize the payload), `AcmePostAsGet` (require an empty payload, else `malformed`), `AcmeOptionalPayload<T>` (`Option<T>`, `None` when empty) — the last for the authorization resource, where one URL serves both a POST-as-GET read and a §7.5.2 deactivation. Supports EC `ES256` and RSA `RS256`. Two mutually exclusive auth forms per §6.2 (both `jwk`+`kid` or neither → `malformed`): an embedded **`jwk`** is verified and re-encoded as DER SPKI by hand (`simple_asn1`); a **`kid`** is resolved to its account and verified against the account's **stored** DER-SPKI, after checking the SPKI's own `AlgorithmIdentifier` OID matches the client's `alg` — so the verification algorithm never rests on `alg` alone. EC coordinates must be exactly 32 octets (RFC 7518 §6.2.1.2), or a short/long one parses as a *different* point, registering one key as two accounts. `SignatureError` (`Malformed`/`BadAlgorithm`/`BadSignature`/`Encoding`) maps to `malformed`(400)/`badSignatureAlgorithm`(400, carrying the `algorithms` list §6.2 requires)/`unauthorized`(401)/`server_internal`(500); an unknown `kid` → `accountDoesNotExist`.
3. **Handler** (`src/handlers/account.rs`) does its own work — `url`/nonce were already checked. `post_new_account` honours `onlyReturnExisting` (§7.3.1: look up, never create), else find-or-creates by `pubkey`, returning `201`/`200` + `Location`; **both branches refuse a `deactivated` account** (`refuse_deactivated`, §7.3.6's MUST — the found branch only, since a just-created account cannot be one), which is what stops a shut-down key confirming its account exists and reading its `contact` list back. `post_account` requires `account.pubkey == pubkey` (ties `kid` to the target it modifies), rejects a `deactivated` account, then applies the update.
4. **Order handlers** (`order.rs`, `authz.rs`) pin ownership through shared helpers. `signer_account` resolves the signer's account and **refuses a `deactivated` account** — every order-side operation funnels through it, so one check covers all seven endpoints. `load_owned_order` requires `order.account_id == account.id` (unknown → `malformed`, wrong owner → `unauthorized`) and refuses an expired order unless already `valid`; expiry is checked *after* ownership so it can't be used to probe order IDs.
   - `post_new_order` normalizes each identifier and validates shape (`well_formed_name`: a `*` is legal only as a single leading `*.`, **and what is left has to be a DNS name** — bounded length, non-empty labels from `[A-Za-z0-9_-]`, no leading or trailing hyphen; `_` stays legal, since underscore labels are ordinary on internal networks). That syntax half is load-bearing: `filter::custom` comma-joins the identifiers into `ACME_FILTER_IDENTIFIERS`, and `challenge::http_01` hands the value to `Url::parse`, which reads `internal.corp/` as the host `internal.corp` — a name the anchored `deny` regex `internal\.corp` does not match, i.e. a deny-list bypass wherever `[filter]` is the only access control. Then policy (`types_for` empty → `rejectedIdentifier` naming `dns-01`), collecting **every** offending name rather than the first, so a multi-name order comes back as one `compound` problem with a `subproblems` entry per identifier (§6.7.1; a lone rejection stays its own type, unwrapped); then `replaces` if present (RFC 9773 §5); then the order plus one `Authorization`/`Challenge` set in **one transaction** — a half-written order would be finalizable for names never authorized.
   - `post_authz` serves both §7.5's read and §7.5.2's deactivation off one URL, told apart by whether a payload arrived; deactivating demotes a `ready` order back to `pending` and is refused once the order is `valid` (revocation, not deactivation, undoes issuance).
   - `post_challenge` refuses an expired *or deactivated* authorization, short-circuits three already-decided states (a `valid` challenge, a terminal `invalid` one, a `pending` sibling on an already-`valid` authorization), else computes the key authorization and calls `challenges.validate`. Both outcomes are recorded in **one transaction** (`commit_validation`/`commit_validation_failure`) — challenge, authorization and order together, with the in-memory sync deferred until after the commit. Success promotes the order to `ready` once every **authorization** (not challenge) is `valid`, and that readiness read happens *inside* the transaction: from the pool, two concurrent validations could each read before the other's write landed and neither would promote, and three separate statements also left a gap where an order sat `pending` with every authorization `valid`, which nothing re-derives. §7.5.2's deactivate-and-demote pair is one transaction for the same reason. **Both outcomes return `200` with the challenge object** (§7.5.1 — a 4xx surfaces as a transport failure to certbot's `acme` library) plus the `Link: rel="up"` header that library requires.
   - `post_finalize` requires `status == "ready"` (else `order_not_ready`), then **claims the order** (`Order::claim_for_finalize`, `ready → processing`, guarded on `rows_affected`) immediately before calling the signer — late rather than at the check, so only the three arms below owe a `release_finalize_claim` rather than every refusal above. A concurrent finalize that loses the claim gets `403 orderNotReady` and polls. On issuance, `BadCsr` leaves the order `ready`/retryable, `Internal` also marks it `invalid` (terminal, pollable). **Everything past the `ready` check is audited**: the success as `certificate_issued`, each of the five refusals below it (bad base64, unparsable CSR, CSR/order mismatch, a filter denial, the backend's own `BadCsr`/`Internal`) as `certificate_issue_failed` with the problem type as `reason`. Above that line refusals are protocol bookkeeping with no CA action attempted, and recording them would bury the ones that matter. The `IssueOutcome::Processing` arm writes **no** row — nothing has been signed — and parks the request's `ClientContext` on the `upstream_orders` row, so the row `signer::relay::relay::settle` writes minutes later can still name the client that asked.

Every one of `post_revoke_cert`'s refusals is likewise an audit row, including the unknown-certificate case — a stream of those is somebody enumerating serials, which is what the trail is for. The two payloads that never reach an identifiable subject (unparsable base64, unparsable DER) write nothing.

**Certificate revocation** (`post_revoke_cert`, §7.6): the JWS need not name an account (either the order's `kid` or the cert's own `jwk`), so it resolves authorization itself rather than through `signer_account`. It decodes the certificate, looks the order up by serial (`find_by_cert_serial`, indexed) and confirms an **exact DER match** against the stored leaf. **Never compare a key re-derived from the submitted DER**: the DER match already proves it equals the stored one, so re-deriving would let anyone who merely observed the certificate on the wire revoke it with an unrelated key. It checks the verified `pubkey` against the order's account or its stored `cert_pubkey`, refuses `alreadyRevoked` (*after* authorization, so an unauthorized caller can't probe revocation state) and an out-of-range reason code, then calls the signer's `revoke` **before** `Order::revoke` — the CA-side action is authoritative, so a signer failure leaves the order un-revoked for a retry. `get_crl` serves the signer's CRL as `application/pkix-crl`; routed but not advertised in the directory (CA infrastructure, not an ACME resource). `get_ca_chain` is its twin at `GET /ca.pem` — the same routing answer, `SignerBackend::ca_chain_pem` defaulting to `None` the way `crl_der` does, `404` for a backend with no anchor of its own (both delegating ones). Media type `application/x-pem-file`, deliberately **not** `application/pem-certificate-chain`: §7.4.2 defines that for an end-entity chain, leaf first, which is the reverse of this.

**Models** (`src/sqlite/`):

- `nonce.rs` — single-use (`DELETE` + `rows_affected == 1`), within `nonce.ttl_seconds`. `cleanup()` runs at startup **and then on an interval for the process's life** — every response mints a nonce, including `GET /newNonce`, so a startup-only sweep would leak.
- `account.rs` — `find_or_create` dedupes by pubkey (§7.3). There is deliberately **no** unauthenticated `GET /acct/{id}`: it would leak `contact` to anyone holding the id, which `newAccount` already publishes in `Location`. `post_account` accepts either `kid` or `jwk`, since the pubkey check authorizes either.
- `order.rs` — `find_by_account` is unfiltered (the admin CLI counts it to say what a delete cascades); `find_active_by_account` is the §7.1.2.1 view the orders-list URL serves; `search` is the **only** implementation of the profile/account/status filter, in SQL (the CLI used to hold a second one in Rust over `list_all`, i.e. one meaning of `--status` written twice and a whole table loaded to filter three fields). `set_pending` is the state machine's one backwards transition, and exists only so §7.5.2 can hold for an order that already reached `ready`; it is bare-statement only, since the demotion commits in the same transaction as the deactivation that caused it. `revoke()` stamps `revoked_at`/`revocation_reason` **without touching `status`** (RFC 8555 defines no "revoked" order status). `to_json` never exposes the revocation columns; that state is admin-visible only (`src/admin/render.rs`). `processing` is now **every** backend's transient finalize state, not only a deferred one: `claim_for_finalize` is the guarded `ready → processing` write `post_finalize` takes *before* calling the signer, and `release_finalize_claim` its guarded inverse for the arms that must leave the order retryable. Two concurrent finalizes on one order used to both pass the `ready` check, both get signed, and leave the loser's certificate in no row at all — unrevocable by `POST /revokeCert`, by the CLI, or by the CRL. `rows_affected` decides; the loser gets §7.4's `403 orderNotReady`. `relay` was already safe (`upstream_orders.order_id` is a primary key), so this closes it for `local_ca` and `custom`.
- `authz.rs` — each `Challenge` gets its own random token (§8), since every key authorization derives from it. `mark_invalid` deliberately doesn't stamp `validated` (§8 defines it as the time of a *successful* validation).

**Wildcard storage convention**: a wildcard authorization's row keeps the wildcard form (`*.example.com`); `to_json` renders the base name plus a separate `"wildcard": true`. Storing the base name instead would break the canonical order `["example.com", "*.example.com"]`: `UNIQUE(order_id, identifier)` compares serialized JSON, both rows would be identical, and the whole order would fail to persist — fixing that would need a `wildcard` column *inside* the constraint, i.e. a full table rebuild. Deriving the base name costs a `strip_prefix` and no migration.

**Signer abstraction** (`src/signer/`): the `SignerBackend` trait — `issue`/`revoke` (async, so a network-delegating backend can await IO; `revoke` must be idempotent), `crl_der` and `renewal_info` (both default to "nothing to say here"), plus three defaulted methods a single backend overrides: `jobs` (the durable-queue handlers this backend needs; it replaced a `resume` hook that re-spawned tasks at startup, so recovery is one case of a queue rather than a mechanism of its own), `http01_tokens` (hands `build_app` the store the `/.well-known/acme-challenge/{token}` route serves from — a getter on the trait rather than a `build_app` parameter, for the same reason `crl_der` is one) and `crl_pruner` (hands `cli::build_generation` the ledger the daily CRL prune sweeps). That last one is deliberately **not** an entry in `jobs()`, and the distinction is load-bearing: `JobRegistry::register` refuses two handlers for one `kind`, and two profiles over *different* `[signer.local_ca]` sections are two backends the `Arc`-identity dedup does not collapse — so a handler returned from `jobs()` would make a supported configuration a startup error. Handing over the *state* lets one `CrlSweepJob` cover every CA in the process. `issue` also takes a `RequestedValidity`, the order's own `notBefore`/`notAfter` (§7.4): a backend may ignore it (both delegating ones do, since the upstream/script decides), but `LocalCa` honours it *clamped* — the request may only narrow its `leaf_validity_days` window, never widen it, and an inverted request is discarded whole. `renewal_info` returns a `RenewalWindow { start, end, explanation_url }` rather than a bare pair, so RFC 9773 §4.2's `explanationURL` survives the relay. `custom.rs`'s `CustomScriptSigner` delegates all four hooks to an external script, over the shared `src/script_hook.rs` hardening.

**`LocalCa`** (`local_ca/`, `rcgen` 0.14) is the default backend. It generates a self-signed CA (`pathLenConstraint: 0`) if none exists; the key is created with `OpenOptions::mode(0o600).create_new(true)`, not chmod'ed after the fact, avoiding a window where any local user could read it. `issue` rejects any SAN that isn't a `DnsName`, requires the DNS SANs to equal the order's identifiers (else `BadCsr`), then **overwrites every extension the CSR asked for** before signing (`sanitize_csr_params`): an empty subject, `is_ca = NoCa` (which omits `basicConstraints` entirely — `ExplicitNoCa` writes the `cA: FALSE` that is the ASN.1 DEFAULT, and certbot's `cryptography` rejects the chain over it), key usages, a fresh random serial, the AKI, `leaf_validity_days` validity, and the two pointer extensions from `LeafPolicy`. That reset matters: rcgen's CSR parser otherwise copies a requested `basicConstraints`/`keyUsage` straight into the signed leaf, so without it a client authorized for one name could submit a CSR carrying `CA:TRUE` + `keyCertSign` and receive a **working intermediate CA**. `revoke` records the serial — and the leaf's own `notAfter`, read with `cert::cert_validity` and best-effort, since refusing to revoke over a housekeeping field would be worse — in a ledger, then regenerates the CRL (idempotent); an empty, validly-signed CRL exists from construction, before any revocation. **Expired entries are pruned** (RFC 5280 §3.3), which is what stops the ledger growing for the life of the deployment; the two rules and the durable `crl_number` it forced are in `../CLAUDE.md` under `signer.local_ca.crl_path`. Three things about the mechanism: the write path is `LedgerStore` (the ledger, the issuer, the paths) rather than `LocalCa`, because `SignerBackend::crl_pruner` takes `&self` and so cannot produce an `Arc<LocalCa>`; `init_ledger` therefore **writes** the sidecar as well as reading it, the counter it just advanced being durable only if written; and a failed persist **restores** the entries it removed, since a ledger holding *fewer* revocations than the file and the served CRL is the unsafe direction of that disagreement. **Where its key lives is pluggable through rcgen's own trait rather than one invented here**: `rcgen::SigningKey` is public and `Issuer<'a, S>`/`signed_by`/`self_signed` are generic over it, so `LocalCa` holds an `Issuer<'static, CaSigningKey>` (`local_ca/key.rs`) — a `Software(KeyPair)` variant plus, behind `--features hsm`, a `Pkcs11` one — and `issue`/`revoke`/`crl_der` never name the key type. Two consequences: the issuer is behind an `Arc` and **both** signing calls (`csr.signed_by` in `issue`, `build_crl` in `revoke`) run under `spawn_blocking` *unconditionally*, since `SigningKey::sign` is synchronous and called deep inside `signed_by` (a token round trip would stall a runtime worker, and branching on the key kind is a correctness trap); and the key type must be `Send + Sync`, which is why `Pkcs11SigningKey` wraps cryptoki's `Send`-but-not-`Sync` `Session` in a `std::sync::Mutex` (the call never awaits, so an async mutex would buy nothing).

**Challenge subsystem** (`src/challenge/`): a trait, an error enum the *caller* maps to `Problem` (`challenge_problem` in `handlers/helpers.rs`), and a `from_config` selector that fails fast at startup. **Bypass is a flag on the registry, not a `NoopValidator`** — `from_config` must not *construct* the real validators when bypassing, since building them reads `/etc/resolv.conf` (or dials `dns.resolver`) and `TlsAlpn01Validator` builds a rustls `ClientConfig`; bypass is the default every test uses. `validate` wraps every attempt in `tokio::time::timeout` at the registry level. One shared `Arc<dyn Resolver>` (`dns.resolver` or system config, **uncached** — a client publishing a `dns-01` record moments before triggering would otherwise be defeated by a cached negative answer) is built once and passed to every validator that needs one; `filter::from_config` makes the same choice for `reverse_dns`.

- **`http_01.rs`** — GET the challenge path, body trimmed and compared to the key authorization. Behind an `HttpFetcher` trait; the production `hyper`-based impl deliberately doesn't add a `Host` header (a loopback test pins this).
- **`dns_01.rs`** — TXT at `_acme-challenge.<name>`, matching `base64url(SHA256(keyAuthorization))` — the digest, not the raw key authorization `http-01` serves. Matches *any* TXT record at the name, since a base+wildcard order publishes two.
- **`tls_alpn_01.rs`** — handshake with SNI = identifier, ALPN `acme-tls/1`. `verify_acme_identifier` requires exactly one matching `dNSName` SAN plus a **critical** `id-pe-acmeIdentifier` (`1.3.6.1.5.5.7.1.31`) extension holding the digest, compared with `subtle`'s constant-time equality.

**Two rustls constraints worth not rediscovering.** (1) The crypto provider is passed **explicitly** via `builder_with_provider`, never `CryptoProvider::install_default` (which panics on a second call and would make test ordering matter). (2) `AcceptAnyServerCert`'s signature-verification methods **assert rather than delegate** to `rustls::crypto::verify_tls*_signature` — those parse the peer certificate with `rustls-webpki`, which rejects an unrecognized **critical** extension, exactly what RFC 8737 requires `id-pe-acmeIdentifier` to be. Proven by the `tls_alpn_01::tests::loopback` test (`UnsupportedCriticalExtension`), whose server also has to resolve its certificate via `ResolvesServerCert` rather than `with_single_cert` for the same reason.

**http-01 redirects are an SSRF surface.** Following them is required by the spec, and Boulder's RFC1918-blocklist mitigation doesn't apply here — serving private networks is the point of this server. What keeps it contained: only `http`/`https`, only the two configured ports, at most `max_redirects` hops, the shared timeout, a `follow_redirects` off switch, and the fetched body is **never echoed into the client-visible error** (only its length; a truncated preview is logged at `debug`).

**Filter subsystem** (`src/filter/`): the `Check` trait (`kind`/`stages` plus two default-pass hooks), a three-valued `Verdict`, and a `from_config` that parses every CIDR, regex and condition at startup (a bad one is fatal) and warns when no rule is configured. `Outcome::Deny`/`Undecided` map to `access_denied`(403)/`server_internal`(500) at the middleware, `rejected_identifier`(403)/500 at `newOrder`, `bad_csr`(400)/500 at `finalize`. There is no `FilterError`: one type, mapped once, at each of the two call sites.

Four laws the evaluator rests on, each a bug if dropped:

1. **Kleene logic**, so an unknown propagates only when it could change the answer (`fail and unknown = fail`, `pass or unknown = pass`). Short-circuiting is structural in the evaluator, not in the combinators, because the hooks are `async` — and an `Undecided` left operand skips *nothing*, since the right operand is exactly what might rescue it.
2. **A rule's stages are the intersection of its checks'**, never the union: evaluating a rule where one check cannot run would substitute a silent `Pass`. An empty intersection is a startup error naming both sides.
3. **Each stage is evaluated independently and both must allow**, and **a stage with no applicable rules allows** without consulting `filter.default`.
4. **A rule whose condition came back `Undecided` is remembered, not skipped.** Once the policy reaches an answer, if that unknown rule's effect differs from the effect actually reached, the whole stage is `Undecided`. Without this, rule *order* would decide whether an outage is survivable.

Per-request **memoisation is mandatory, not an optimisation**: a check named twice runs once, because `custom` forks a process and `ipam` makes four HTTP requests.

- `ip_allow.rs`, `reverse_dns.rs` (PTR + optional forward confirmation + hostname regex, behind a `Resolver` trait with a timeout budget) and `identifiers.rs` (type then regex) share `allow`/`deny` semantics via `filter::check_lists`: **`deny` wins**, checked first; an empty `allow` imposes no constraint (deny-only = a working blocklist); membership, not longest-prefix-match. `allowed_ip` errors at startup if both lists are empty (a pure no-op); the other two stay meaningful with empty lists. All fail closed on a missing client address (`ConnectionContext::require_client_ip()`), including in blocklist mode — an address the server can't see isn't "absent from deny, therefore fine". `reverse_dns` applies `deny` across **every** PTR candidate, not just the one that would otherwise be accepted.
- **CSR identifiers are typed and flattened** (`csr_identifiers`): SANs plus a subject `CommonName` all project into one `Identifier{typ, value}` list (`dns`/`ip`/`email`/`uri`/`other`/`cn`), so a deny rule can't be dodged by moving a name between them. `deny` reaches every type including `cn`; `allow` skips `cn` (`SUBJECT_ONLY_TYPES`), since a CN is legacy subject metadata and routinely a human label (rcgen defaults to `"rcgen self signed cert"`), not a name the certificate is *for*.
- Regex patterns are auto-anchored (`^(?:…)$`, case-insensitive) — the `regex` crate searches rather than matches, so an unanchored `example\.com` would also accept `example.com.evil.net`.
- **Client IP** (`client_ip.rs`): `ProxyPolicy::resolve` trusts the peer unless it's in `trusted_proxies`, then walks the forwarded-for list right-to-left past trusted hops. Addresses are canonicalized first (`IpAddr::to_canonical()`), since the dual-stack `[::]:3000` bind sees IPv4 clients as `::ffff:…`. Requires serving with `into_make_service_with_connect_info::<SocketAddr>()` or filters have no peer address at all and fail closed.

**HTTPS termination** (`src/tls.rs`): `from_config` returns `None` — touching no disk — when `server.tls.enabled` is off. Otherwise it loads or generates+writes a self-signed cert for the `base_url` host, then reads back exactly what was written. The rustls `ServerConfig` passes the crypto provider explicitly (same rule as above) and advertises `http/1.1` only. `TlsListener` implements `axum::serve::Listener`:

- **Handshakes run off the accept path.** A background task accepts TCP and spawns each handshake under `handshake_timeout_ms` into a bounded channel `accept()` drains (the slot is reserved before accepting — both the backpressure and how a dropped listener stops the task). Doing the handshake inline would let one stalled client block every other connection for the length of the timeout.
- **`TlsListener::spawn` returns `HttpsListener = TapIo<TlsListener, …>`.** `into_make_service_with_connect_info::<SocketAddr>()` needs `SocketAddr: Connected<IncomingStream<'_, L>>`, which axum implements only for the concrete `TcpListener` and — blanket — any `TapIo<L, F>`. The missing impl can't be written here (foreign trait, foreign type — coherence refuses it), so `TapIo` carries the peer address into request extensions; without it, IP filters see no client address under TLS and fail closed.

## Module layout

- `src/main.rs` — the binary entry point, and the **only** place in the project that prints to stderr and calls `std::process::exit`. Parsing argv, loading the configuration, installing the subscriber and opening the database all live here rather than in `src/cli/`, because nothing that links this crate can use a function whose failure mode is ending the process. That is also why it is excluded from the coverage floor (see CI): all four of its failure branches are unreachable from a test.
- `src/cli/` — the `clap` command tree, plus `style.rs`/`render.rs` and `logging.rs` (`[logging]` → an installed subscriber, and the handle a reload swaps it through: `prepare_logging`/`publish_logging` are the build/publish pair, and three things that module's shape rests on are documented there — the filter is composed with `and_then` and **never** `with_filter` (`reload::Handle::reload` is unusable with a `Filtered` layer), the whole stack is **one** boxed layer because `Box<dyn Layer<S>>` must name its `S`, and the format layer goes on the *inside* because `Layered::max_level_hint` directly over a `Registry` returns the outer hint alone — the readable order silently pins `LevelFilter::current()` at `TRACE`. The handle is a `OnceLock`, beside the global it is a handle to; unset, `publish_logging` is a no-op that reports `false` rather than claiming a swap). **No command body prints or exits**: every one returns `Result<(), CliError>` and `dispatch` routes to it, so each arm is a plain function a test can call rather than an unreachable dead end. `src/main.rs` is where that `Result` becomes an exit status — the whole reason it is the binary and not the library. `CliError` carries the message and has `From<sqlx::Error>` (`"database error: …"`), which is why the DB calls are plain `?`. `style.rs` is the colour decision and vocabulary (`ColorChoice`, `Palette`, hand-rolled SGR — `src/metrics.rs`'s no-dependency reasoning), `render.rs` the human-readable renderings it paints. **Those renderers moved here out of `src/admin/render.rs`**, which is the front-end-agnostic layer and where a `Palette` argument would not belong: every `render_*_json` is a wire format two front ends parse, while every `render_*_line`/`render_*_text` and `print_rows` has exactly one consumer, the terminal. `dispatch` takes the `--color` *choice* and resolves the palette itself, since the answer needs `stdout().is_terminal()` and `NO_COLOR` and `main.rs` sits outside the coverage floor; it then threads the `Palette` (a `Copy` `bool`) into every runner. Two invariants: `Palette::plain()` returns its argument untouched, so the plain output is byte-identical to what it always was, and `--json` is **structurally** unreachable from a palette. Colour wraps an already-padded field, never the other way round — a format width counts bytes. `serve` binds `server.bind_address` and hands the socket to **`serve_on(config, database, listener, shutdown)`**, which runs `webadmin::check_config`, binds the admin socket when `[admin]` is enabled, and delegates to **`serve_on_with(…, admin_listener, …)`**; the split lets a test drive the two-listener path on two ephemeral ports. `serve_on_with` does everything else — profile resolution, deduplicated signer backends, per-profile filters/challenge validators, TLS, the `JobRegistry` assembled from `SignerBackend::jobs`, the job runner, the `Auditor` (built here rather than in `Profile::build_all`, since `[audit]` is process-wide), and `axum::serve` with `into_make_service_with_connect_info::<SocketAddr>()`. Splitting on the socket boundary is what lets a test drive the whole startup path on an ephemeral port with its own shutdown future instead of a process signal; only `serve()`'s four-line shell and `shutdown_signal()` stay untested (the process wiring above them having moved to `main.rs`). `cli::build_generation` plus `cli::prepare_reload`/`cli::publish_reload` are the reload *policy* (see `src/reload.rs`), a generation being startup's own assembly re-run. Three traps: the "register once per backend" list holds `usize`, not `*const ()` (a raw pointer held across an `.await` would make the spawned future `!Send`); the `watch::channel` carrying `shutdown` is created *before* profile assembly, so a signal arriving during startup is not ignored, and the job runner takes a receiver like both listeners; and the relay task sits under `AbortOnDrop` so an error path does not leak a task parked on a signal that never arrives. All three `axum::serve` futures now have **one** type — a `listener::RoleSocket` is the same whether its role is speaking TLS, speaking cleartext, or holding no socket at all — so the boxed `type Serving` and the `std::future::ready(Ok(()))` placeholder for a disabled listener are both gone; `try_join!` joins three real futures that live for the process. `Role` is an enum rather than the `&'static str` the log field wants, so every point that decides something per listener is exhaustive. Logic beyond dispatch lives in `src/admin/`.
- `src/lib.rs` — library root: `Profile`, `AppState`, `build_router` (one profile), `build_app` (root routes + one nested router per profile), `metrics_app` (the third listener's router — one route, no admission control, no nonce, no security headers, and deliberately **not** behind a `reload` swap cell, since its only state is a registry that is carried rather than rebuilt), and two structs the reload path rests on: `crate::Assembly` (what survives a generation — the pool, the queue, the metrics registry, the notifier handle, and the previous generation's `SignerSet` behind a `Mutex`, kept only so the next reload can reuse a backend and adopt a rebuilt one's state) and `crate::Egress` (the resolver, the proxy policy and the *rendering* of the `[dns]`/`[proxy]` sections they came from, rebuilt per generation — the identity travels with the clients so the two can never disagree, which is the whole of what makes `dns.resolver` reloadable). `Profile::build_all_with` takes a `GenerationParts` (egress, dispatchers, signers) rather than reaching into the `Assembly`, because all three are built before anything is published and swapped together. Also installs the response-hardening layers (HSTS, `X-Content-Type-Options`, `X-Frame-Options`) and a `GlobalConcurrencyLimitLayer::new(100)` — note it *queues* rather than sheds, so it bounds concurrent work but doesn't reject a flood.
- `src/handlers/` — one file per ACME resource (`account.rs`, `authz.rs`, `certificate.rs`, `directory.rs`, `order.rs`, `renewal_info.rs`), re-exported flat from `mod.rs`; `helpers.rs` holds the shared ownership helpers, the filter-facing CSR projection and the name-shape validators.
- `src/config/` — `mod.rs` (`Config::load`, `LIST_KEYS`, `empty_string_is_no_values`) and `types/` (one file per TOML section: `server`/`signer`/`filter`/`challenge`/`notify`/`profile`, re-exported flat so no import outside the directory names a submodule).
- `src/audit/` — the audit vocabulary (`AuditEvent`, `Actor`/`ActorKind`, `AuditRecord`, `ClientContext`), the `RequestContext` extractor, the cached PTR resolver, and `write`/`Auditor::record`. Deliberately **not** merged into `src/notify/`, which fires at nearly the same call sites with nearly the same fields: a notification is outbound and best-effort (retried, ultimately abandonable), an audit row is inbound and durable and exists for the events nobody wants a notification about — the refusals. `write` is a free function as well as a method because the relay settles from a background task holding an `Arc<Database>` and no `Auditor`, and needs no resolver there (the address was resolved during the finalize request and parked on `upstream_orders`). `Auditor::with_metrics` is the builder step the serving path uses to attach the Prometheus registry: `record` counts the same `AuditRecord` it is about to store, so "how many certificates did we issue" answers identically whether it is asked of `/metrics` or of `audit list`. A builder step rather than a `from_config` argument because `[audit]` and `[metrics]` are independent, and every test wanting an auditor should not have to build a registry. The relay's two `write` call sites bump the counter explicitly for the same reason the free function exists at all — no `Auditor` reaches that task.
- `src/tls.rs`, `src/pemfile.rs`, `src/error.rs`, `src/cert.rs` — as described under Architecture above. `tls.rs` is **provisioning only** (`from_config`/`admin_from_config`/`acceptor_from`/`generate_self_signed`/`TlsSettings`); accepting connections is `src/listener.rs`'s. `pemfile::write_atomic`'s scratch name is `<path>.<pid>.tmp`, and **both halves are load-bearing**: it was `with_extension("tmp")`, which gave `ca.crl` and `ca.json` the same `ca.tmp` (so persisting the ledger could rename CRL PEM into the sidecar — the next startup then refuses to parse the whole revocation history), and which two writers truncate and fill in turn before each renames the mixture into place, atomically wrong. A crash now leaves inert litter instead of a shared mutable temp, which is the right way round.
- `src/extractors/` — `acme.rs` (the `verify_jws` core + the extractors), `jws.rs` (wire types), `signature.rs` (`verify_signature_and_get_der`, `verify_signature_with_spki`, `spki_parts`/`spki_to_jwk`, `jwk_thumbprint`, `SignatureError`).
- `src/eab.rs`, `src/key_change.rs` — pure verification, no database access; the DB lookup and active/revoked filtering happen in the handler.
- `src/signer/mod.rs`, `src/signer/custom.rs`, `src/signer/local_ca/` — as described above, plus the reload seam: `CarriedState` (resource → `Arc<dyn Any + Send + Sync>`), the defaulted `SignerBackend::carried_state`, `SignerParts` (the dependencies minus the `[signer]` section — a struct for `ProfileParts`' reason, since `from_config` had reached seven positional parameters) and `SignerSet` (a generation's backends in two views: by profile name, which serves, and by configuration identity, which is what the next reload compares against). `custom` carries nothing; `local_ca` carries its `Arc<Mutex<RevokedLedger>>` keyed on `crl_path`, `relay` its `Arc<MemoryTokenStore>` keyed on `account_key_path` — and `Inner` keeps that store *concretely* beside `ChallengeStrategy::Http01`'s trait object, because `CarriedState` can only hand back a sized type. `local_ca/` is a directory since the key source became pluggable: `mod.rs` (issuance, and `impl LocalCa` whole), `ca.rs` (the CA's own certificate and the serials it issues under), `crl.rs` (the revocation ledger, the CRL built from it and the sidecar both are persisted to — sharing only an `Issuer<'static, CaSigningKey>` with issuance; also `LedgerStore`, the write path, which is what `crl_pruner` hands over), `sweep.rs` (`CrlSweepJob`, the daily prune — deliberately not in `jobs/sweep.rs`, whose `SweepTarget` is four `DELETE`s needing nothing but a `Database`, where this holds signer state, signs with the CA key and touches no database at all; it keeps that module's two laws, `Reschedule` and **never `Failed`**), `key.rs` (`CaSigningKey`, `KeySource`, `read_pin` — always compiled), `pkcs11.rs` (`#[cfg(feature = "hsm")]`), and `policy.rs`: `LeafPolicy` and the two extensions naming this CA's own services, decided **once at startup** (built in `load_or_generate` before either branch touches disk) so a bad URL is a startup error rather than a 500 at finalize and `issue` does no ASN.1 work per request. Credentials in the URL, a non-`http(s)` scheme and any value `Url` would normalize (a missing trailing `/`, an environment list's un-trimmed `a, b`) are each refused naming the key and the value; `http://` is unwarned on purpose, since fetching a *signed* CRL over TLS is the validation loop this extension breaks. rcgen has no AIA support, so that half is a `CustomExtension` whose DER is built with `simple_asn1`; `id-ad-ocsp` is never written, there being no responder.
- `src/signer/relay/` — the relaying backend, split by concern: `mod.rs` (`ChallengeStrategy`, `PollConfig`, `Inner`, `RelaySigner`, `from_config`, the `SignerBackend` impl — whose `issue` opens the upstream order synchronously then *enqueues* rather than spawning, and whose `jobs()` hands over the one `RelayJob`); `account.rs` (provisioning this proxy's upstream account and the `kid` sidecar — the one part with a lifecycle of its own, run once and only *used* thereafter, and what makes an upstream EAB credential single-use); `wire.rs` (all six upstream DTOs plus `parse_rfc3339`/`upstream_to_signer_error`); `flow.rs` (the background state machine: `RelayJob` — the `JobHandler` — plus `RelayFailure`/`classify`/`relay`/`answer_dns01`/`answer_http01`/`answer_bypass`/`poll_until`/`settle`, everything after `issue` answered `processing`; `classify` is where an `UpstreamError` becomes retryable or permanent, table-tested one row per variant, `settle` returns the job's outcome rather than writing a failure itself, and `RelayJob::abandon` is the old `fail`); `client.rs` (a hand-written outbound ACME client); `testsrv.rs` (a `cfg(test)` scripted ACME server on loopback, so the client is tested against real HTTP without the network — with `Script::http01_responder` set it really *fetches* the challenge file rather than just counting the trigger). The tests enter through `SignerBackend::issue` and assert a *relay* outcome, so they stay in this module, but live in `tests/` beside it (`lifecycle`/`eab`/`dns01_strategy`/`http01_strategy`/`renewal`, fixtures in `tests/mod.rs`) rather than as one 2000-line inline block. `dns01.rs` is the one place that **writes** DNS: RFC 2136 + TSIG on `hickory-proto` (not `hickory-client`, whose 0.26 line is still pre-release). `http01.rs` is its twin and the one place that **serves** a challenge file: a `TokenStore` seam over a `MemoryTokenStore` plus a `PublishedToken` `Drop` guard — retraction can run from `Drop` here where the dns-01 side's network round trip cannot, which stops the runner's per-attempt timeout leaking a live key authorization per timed-out relay. The key authorization at the upstream uses **this proxy's own thumbprint** for both, never the end client's (different accounts on different servers), and http-01 serves it *verbatim* where dns-01 publishes its SHA-256 digest (§8.3 vs §8.4). Two gotchas: `from_config` provisions on a scoped OS thread with its own runtime, since its caller (`cli::serve`) is already inside one and a plain `block_on` would panic; and unlike `challenge::tls_alpn_01`, this client **validates** the upstream's TLS certificate against `webpki-roots` — here the certificate is the only thing identifying the CA being handed CSRs.
- `src/challenge/mod.rs`, `http_01.rs`, `dns_01.rs`, `tls_alpn_01.rs` — as described under Architecture above.
- `src/dns.rs` — the shared `Resolver` trait + `HickoryResolver`, used by every DNS-touching consumer, so `dns.resolver` genuinely governs every outbound lookup: `Profile::build_all` builds one uncached resolver and threads it through `signer`/`notify`/`filter`/`challenge`. `reverse_dns` deliberately keeps building its own **cached** resolver from `DnsConfig` — a PTR lookup for an address that keeps connecting is what a cache is for, while the shared one must stay uncached so a `dns-01` record published moments before a trigger is not defeated by a cached negative.
- `src/http_client.rs` — the transport half of the four outbound HTTP clients: `Endpoint::from_url` (host/scheme/port), `connect` (resolve → optional `CONNECT` tunnel → TLS → `http1::handshake` → detached connection task), `connect_stream` (the same minus HTTP, for `tls_alpn_01`'s raw probe) and `webpki_tls_config`. Deliberately **not** a shared client type: each caller keeps its own headers, body cap, error type and — the part that actually differs — its own TLS *choice*, since `challenge::http_01` must not validate the responder's certificate (RFC 8555 §8.3) while the other three must. An `https://` target is reached by a **`CONNECT` tunnel**, an `http://` one is *forwarded* (absolute-form request line + `Proxy-Authorization`), and `tls-alpn-01`'s raw probe is tunnelled too. `connect` returns a **`Connection<B>`**, not a bare `SendRequest`, because two things are properties of the *connection* rather than the caller: `request_target(&Url)` (absolute-form only when forwarding through a proxy, origin-form otherwise, so a request built for one shape is never sent on the other) and `send_request`, which attaches `Proxy-Authorization` and **never inside a tunnel**, where the credential was already spent on the `CONNECT` and repeating it would hand it to the origin. Three gotchas: `Endpoint::connect_authority()` always carries the port where `authority()` elides a default one (`CONNECT example.com` is malformed and every real proxy 400s it); the tunnel's `hyper::client::conn::http1::Connection::with_upgrades()` is load-bearing and its absence **hangs** rather than failing, which is why three loopback tests are shaped to time out; and the status is checked *before* `hyper::upgrade::on`, since a non-2xx has no upgrade to hand over and its body ("407 Proxy Authentication Required") is the whole diagnosis.
- `src/proxy.rs` — the policy half of the above: `OutboundProxies` (`from_config` with the environment fallback, `select`, `direct`), `ProxyTarget` and the `no_proxy` rules. Separate from `http_client` because that module owns "the plumbing and nothing else", and this is a fallback precedence, six matching rules, a refusal vocabulary and a credential. `ProxyTarget`'s `Debug` is hand-written and `redacted()` is the only rendering of the URL that exists, so a password never reaches a ticket. `OutboundProxies::always` (`#[cfg(test)]`) exists because the unconditional loopback bypass would otherwise make every transport test against a proxy on `127.0.0.1` take the direct path and pass for the wrong reason. Reuses `filter::{parse_net, canonical}` rather than growing a second CIDR parser; the module doc names the hoist-to-a-neutral-module option for whoever adds a third consumer.
- `src/filter/mod.rs`, `client_ip.rs`, `ip_allow.rs`, `path.rs`, `reverse_dns.rs`, `identifiers.rs`, `eab.rs` — as described above. `path.rs` replaces `filter.exempt_paths` and exists for one concrete trap: **`/crl` is served by the profile router**, so an address-based policy without a path rule silently breaks revocation checking for every relying party outside the allowlist. Its `*` stops at `/` (one path segment), deliberately *not* sharing `glob_to_pattern` with the name checks, whose `*` stops at `.`. `eab.rs` matches the label of the credential an account registered under — the multi-tenant lever, writeable in advance where an account id is not; it needs no migration, since `accounts.eab_kid` has recorded this since EAB was implemented, and promotes that column from audit trail to policy input.
- `src/filter/policy.rs`, `expr.rs`, `build.rs`, `explain.rs` — the engine. `policy.rs` holds `Check`/`Verdict`/`StageSet`/`FilterPolicy` and the evaluator; `expr.rs` the condition parser (hand-written recursive descent, column-bearing errors, a `Display` that re-prints with explicit parens — what `filter show` prints); `build.rs` the config→policy resolver, **the only place that knows which flattened key belongs to which type**, and where every startup refusal is worded; `explain.rs` everything the two CLI commands print.
- `src/filter/ipam.rs` — the `ipam` check, `src/ipam/`'s only consumer. A **second** `ipam` check in one profile is a startup error: a profile has exactly one inventory, the one `[ipam]` configures. Identifier hook only (a connection hook would query the inventory on every `newNonce`). Holds nothing but an `Arc<IpamRegistry>` and interpolates `backend_name()` into every refusal, so a phpIPAM operator reads "phpIPAM" in the 403. Matching is exact (case/trailing-dot-insensitive, otherwise literal — `*.example.com` needs that exact string).
- `src/ipam/` — the inventory subsystem. `mod.rs` (`Ipam`, `AddressNames`, `IpamError`, `Source`/`parse_sources`, `IpamRegistry` and its budget, `from_config`, plus the shared `normalize`/`field_values`/`value_kind`), `http.rs` (the JSON-over-HTTP transport **both** backends share — the one case `http_client`'s "policy stays per-module" rule does not cover, since these two have the same policy; `JsonApiError` carries the status because that is the only thing that differs), `netbox/{mod,client}.rs` and `phpipam/{mod,client}.rs`. Each backend keeps a trait seam over its own queries (`NetboxApi`/`PhpIpamApi`) so the policy is testable without a server, the shape `crate::dns::Resolver` gives `reverse_dns`. **`sources`** names where a permitted name may come from (`dns_name`/`custom_field`/`device`/`vip`/`fhrp`): `device` is a fallback (read only when the address said nothing), `vip` and `fhrp` are unions. **`fhrp` is a membership proof and the direction of the query is why**: client address → its interface → `fhrp-group-assignments?interface_id=…` → group ids → their addresses. Nothing is ever looked up by group name, by the service address, or by the requested identifier, so no query can reach a group the client is not recorded in. The budget lives on `IpamRegistry` (one `timeout_ms` covering all four-or-five requests) the way `ChallengeRegistry`'s does, so a backend added later cannot forget it.
- `src/jobs/` — the durable queue. `mod.rs` (`JobOutcome`, `JobHandler`, `JobSpec`, `JobQueue` — the enqueue side plus the runner's `Notify`), `registry.rs` (`JobRegistry`; a duplicate `kind` is a startup error, since two handlers would each claim about half the rows), `runner.rs` (`spawn_runner`, the claim loop, `backoff`, the graceful stop), `sweep.rs`. The loop watches **two** `watch` cells, the registry and `[jobs]` itself, and reads both at the **top of a pass** rather than in the arms that wake on them — an arm can only apply what it caught, and a pass is entered from five different wake-ups. `Permits` wraps the semaphore because it has no `resize`: growing lands whole, shrinking can only take back *free* slots and so converges as jobs finish, and its `capacity` (what is really issued, not what is configured) is what the graceful stop drains against. Two traps encoded there: `drain_ready` uses `try_acquire_owned` and reports saturation instead of waiting, since parking on a permit made the loop deaf to a reload *and* to shutdown under a backlog; and an arm whose `watch` sender has been dropped must be **disabled** via `has_changed().is_ok()`, because `changed()` on a closed channel returns `Err` immediately and for ever — a permanently ready arm, i.e. a spin, which is what `spawn_runner`'s two fixed cells used to cause. The model is `src/sqlite/job.rs`; `Job::count_live(kind)` is the kind-wide counterpart to `find_live`, for callers with no single `dedup_key` to ask about (a `notify_deliver` key is a per-occurrence uuid). **No `#[instrument]` anywhere in this module**, the rule `src/webadmin/` keeps and for both its reasons.
  - Claiming is a guarded `UPDATE … RETURNING` (the `admin_sessions.mfa_attempts` shape) taking a **lease**, so a crashed process's work is reclaimed by expiry rather than only by a restart, and two runners over one database cannot both take a row. `attempts` increments at *claim*, so a job that kills the process still exhausts its budget.
  - Retrying is bounded twice: by `jobs.max_attempts` (frozen onto the row at enqueue) and by the row's own `deadline`, which the relay sets to the local order's `expires` — past that the order is refused on read, so a certificate obtained upstream could never be collected.
  - `enqueue` wakes the runner through a `Notify` rather than leaving it to `jobs.poll_interval_ms`, because the ACME client that triggered the work is already polling its order.
  - Two ordering traps are documented at their call sites: the concurrency permit is acquired **before** the claim (claiming first would start every backlogged row's lease ticking while it queued, and each would then be reclaimed as "crashed" having never run), and `abandon` is called only *after* the settling write took, so a runner whose lease was reclaimed does not tell a subject its work failed while another runner is still doing it.
  - `sweep.rs` holds **all four periodic table sweeps** over one `SweepTarget` enum (nonces, `audit.retention_days`, admin sessions, the queue's own `retention_days`) — the `Reschedule` worked example: periodic work without a cron table, one row that re-queues itself, whose identity the partial unique index keeps unique. Three of them used to be `spawn_*_reaper` interval loops (with `retention.rs` holding the fourth, which was already a job), near-copies of each other. Moving them buys three things and costs one: a sweep whose task died is reclaimed by lease expiry instead of being silently gone until the next restart; the schedule survives a restart, so a server restarting more often than once a day no longer skips the daily sweeps for ever; there is one answer in the process to "run this every N seconds"; and the sweeps now stop if the runner does. `recover` **is** the startup sweep — it enqueues at `run_at = now`, so the runner performs the first pass on its way into the loop and the explicit `Nonce::cleanup`/`AdminSession::cleanup` calls in `serve_on_with` are gone. The event names are unchanged (`nonce_reaper_swept`, `audit_reaper_swept`, `admin_session_reaper_swept` + `_failed`) because `doc/src/operations/monitoring.md` names them and `tests/logging_convention.rs` checks it. **A sweep never returns `Failed`**: a retired periodic job does not re-enqueue itself, so one transient database error would stop that sweep for the life of the process.
- `src/notify/` — the backends (`email`, `webhook.rs`, `custom`), `NotifyDispatcher`, `NotifyEvent`/`NotifyError`, `build_environment` + the embedded `webhook/<event>.j2` templates, and `job.rs` (the `NotifyJob` handler the queue runs a `notify_deliver` row through). Behaviour and the retryable/permanent split are under What this is in `../CLAUDE.md`.
- `src/reload.rs` — the reload mechanism: `SwapService`/`swappable`/`router_channel` (the router cells), the `FROZEN` projection table and `check_frozen`, `ReloadError`/`ReloadReport`, `channel()`/`ReloadHandle`/`Reloads`. The *policy* — what one generation is and in what order it is published — lives in `cli::build_generation` and the `cli::prepare_reload`/`cli::publish_reload` pair, split so the fallible half can run on `spawn_blocking` (building a `relay` backend contacts its upstream) while the publishing half keeps its uninterruptible, await-free run. `Reloads::none()` is a channel whose sender is already dropped, which makes every caller that serves no reloads free rather than a second code path. What it rests on:
  - **`watch::Sender::send_replace` is synchronous, and that is the whole atomicity argument**: a run of sends with no `.await` between them cannot be interleaved, so nothing observes a half-swapped generation — no lock, and no `arc_swap` dependency.
  - The publishing order has one constraint, **notifiers and the registry before the routers**: a request served by the new generation queues a `notify_deliver` row naming a slot id from the new configuration, and a `NotifyJob` still holding the old map would retire it *permanently* (an unknown backend id is `Failed`, not `Retry`).
  - The routers sit behind a `SwapService` used as a **`fallback_service`, not a make-service** — a make-service is consulted per *connection*, so an HTTP/1.1 keep-alive client would hold the old router for its lifetime. Nesting means two `top_level` route futures, which is idempotent (`set_content_length` returns early, `set_allow_header` no-ops, stripping an empty HEAD body does nothing), pinned by a `HEAD` regression test.
  - `FROZEN` is a projection table (`fn(&Applied) -> String`, since nothing in `src/config/` derives `PartialEq` and `Debug` is already this crate's config-identity primitive) — a whole-section compare could only say "server changed". **It is down to one entry, `database.url`**, and that is the only one ever frozen *physically* rather than by ownership: the pool is open and the accounts and orders issued against it do not follow a URL elsewhere. Everything else came off by being made movable — `[logging]` behind a `reload::Layer` handle, the seven listener keys once `src/listener.rs` owned the accept loop, the seven `[jobs]` ones once the runner stopped snapshotting its pacing, and finally `profiles`/`profiles.*.signer`/`dns.resolver`/`proxy` through `signer::CarriedState`. `Applied::profiles` survives with no reader in the table, since the pairing is the shape of a resolved configuration and the alternative is a caller reassembling it the moment an entry needs it. **One rule outlived its code and must be reapplied if an entry is ever added back**: `proxy` and `profiles.*.signer` used to render through `opaque` (a truncated SHA-256) because both reach a credential — a proxy URL's `user:password@`, the HSM PIN, the TSIG key, the upstream EAB secret — and `ReloadError::Frozen` embeds both renderings in a message the supervisor logs. A whole-section projection is opaque iff any field it reaches can hold a credential; the comparison only ever needs equality, so a digest costs nothing, and a redacting `Debug` was never available since `format!("{cfg:?}")` of a signer section is the identity key `build_backends` keys on.
  - **A socket is a plan, not a cell.** `cli::plan_sockets` decides `Keep`/`Serve`/`Close` per role by comparing the *configured* addresses old-vs-new — never against what is actually bound, since `serve_on_with` callers legitimately supply a socket that is not `bind_address` (every test binding `127.0.0.1:0` does) and rebinding it out from under them would break the feature's own callers. Every bind happens in the build phase, so a port already in use refuses the reload with the live socket untouched; the publish phase only hands the already-bound listener over, which `mpsc::UnboundedSender::send` does synchronously and so fits inside the same uninterruptible run as the routers. Two addresses that differ as strings but collide in the kernel (`[::]:3000` against `0.0.0.0:3000`) fail that bind, which is the safe direction. `announce_admin_listener` is deferred to the supervisor because it reaches the database, and this function has no await point to spend.
  - **The signer freeze was about state, not side effects — and the state was made portable.** A CA is generated only when absent and a relay registers once, so construction repeats nothing destructive; what was unmovable was that `LocalCa` rebuilds the whole CRL from an in-memory ledger and a relay's `http-01` `MemoryTokenStore` would come back empty under a live upstream fetch. `signer::CarriedState` is the seam — a map from **resource** (`local_ca.ledger:<crl_path>`, `relay.http01:<account_key_path>`) to `Arc<dyn Any + Send + Sync>`, filled by `SignerBackend::carried_state` and read by each backend's constructor. Keyed on the resource rather than on the profile or the configuration, so a rebuild over the *same* files finds the live cell and one over *different* files finds nothing; the keys cannot collide because `signer_paths` already refuses two live backends over one path. `LocalCa.revoked` became an `Arc<Mutex<_>>` for this, and the adoption **shares** it rather than copying — which is what closes the window the durable sidecar cannot: a revocation landing on the outgoing instance after the incoming one read the file and before the swap. `build_backends` reuses a backend verbatim when its identity is unchanged, so most reloads construct nothing at all (and `key_source = "pkcs11"` does not log in to the token again); the identity is `format!("{cfg:?}")` **plus `Egress::identity`**, which is what makes a `dns.resolver` or `[proxy]` edit reach the signers and is therefore why those two keys reload.
  - `LoginLimiter::rebuilt` moves the failed-login buckets into the new limiter under the new limits — carrying the limiter whole would make `admin.login_*` silently stale, rebuilding it empty would hand an attacker a fresh budget per reload. `profile_mounted` deliberately does **not** re-fire: it is a lifecycle notification, not a heartbeat. `watch_for_hangup` does not consume its stream (a one-shot handler would leave the second `SIGHUP` at its default disposition, *terminate*), and is installed in `serve` before anything slow for the same reason.
- `src/script_hook.rs` — the contract all three `custom` hooks run under: `ScriptHook::new/run/detail`, `ScriptStdin::{Null,Json}`, `ScriptError`, `ScriptOutcome`. Owns the hardening (`env_clear()`, minimal `PATH`, `kill_on_drop(true)`, `tokio::time::timeout`) that used to be written out three times token-for-token. Each subsystem keeps only what is its own — env vars, stdout parsing, error mapping, `signer`'s reserved exit code 3, `filter`'s `Denied`-on-nonzero and `pass_stdin`. `ScriptOutcome::stdin_error` records an `EPIPE` on the stdin write and surfaces it through `detail()` *only when the script also failed*, so "the script never saw the CSR" stops reading like "the script rejected the CSR" without breaking a script that ignores its payload.
- `src/testutil.rs` (`#[cfg(test)]`) — `TempDir` (implements `AsRef<Path>`, so it drops into anything taking a path), `write_script`, `identifiers`/`dns_identifiers`, `account_id`/`account_seen_from`/`client_context`, the four row fixtures (`order_fixture`, `audit_entry`, `admin_user_fixture`, `admin_session_fixture` — hoisted when the renderer tests split across `src/admin/render.rs` and `src/cli/render.rs`, which both need them), and `SpanFields`/`capture_request_span` (a `tracing_subscriber::Layer` collecting one named span's fields — the only way to assert a *span* field, since unlike an event field nothing in the response or a log line says whether it was recorded; it collects `on_new_span` **and** `on_record`, or every deferred field would read as absent). Each had grown copies across the crate — seven of `TempDir`, three of `account_id`, twelve hand-rolled `Identifier` builders before `Identifier::dns`/`::new` went into production where they belong. `tests/common/mod.rs` carries a second copy of the directory helpers, because an integration test cannot see a `#[cfg(test)]` item of the crate it links against.
- `src/filter/custom.rs` — the `custom` check (`CustomScriptFilter`): runs an operator script at both hooks, context via `ACME_FILTER_*` env vars and optional stdin JSON; exit 0 permits. `ACME_FILTER_CHECK_NAME` tells it which instance invoked it, so one script can serve several. The child runs with `env_clear()` (this process's environment holds secrets, e.g. the RFC 2136 TSIG key) plus a minimal `PATH`, and `kill_on_drop(true)` (a `tokio::time::timeout` only drops the future, so without this a timed-out script would outlive its deadline and leak a process per request).
- `src/middlewares/filter.rs` — the connection-filter layer; inserts `ClientIp` into request extensions **and records it onto the `request` span**, overwriting the peer address `access.rs` seeded there, since this is the first layer that knows the per-profile `filter.trusted_proxies`. That record is why this middleware carries **no `#[instrument]`**: the attribute opens a child span, `Span::current()` inside it is that child, and a record naming a field only the `request` span declares is silently dropped — the exact bug `access.rs`'s module doc describes `request_id` having had. Note this is the opposite convention from `verify_jws`, which declares `alg`/`account_id` on its *own* `#[instrument]` span and records to that. `src/middlewares/nonce.rs` — the `Replay-Nonce` response middleware. `src/middlewares/index_link.rs` — the `Link: <…/directory>;rel="index"` response middleware (§7.1, on every resource but the directory); it **appends**, never sets, so `post_challenge`'s `rel="up"` survives alongside it.
- `src/listener.rs` — the sockets, and replacing one while it serves. One `axum::serve` per role (`acme`/`admin`/`metrics`) lives for the process, over a `RoleListener` that owns the accept loop: `spawn` returns the `RoleSocket` to serve and the `ListenerHandle` a reload writes through. Two things are swappable underneath, both read **per connection** — the `TcpListener` (through an unbounded mpsc, so the send is synchronous) and an `Option<TlsSettings>` (so `tls.enabled` is a mode, not a listener type: **a flip on an unchanged address rebinds nothing at all**, which is the one case a bind-then-drain scheme could not serve, two listeners not being able to hold one port). `MaybeTls` is the shared `Io` that makes one `axum::serve` outlive that flip; the TLS half is boxed, since the enum is as wide as its largest variant on every cleartext connection too. `bind_blocking` is `std::net::TcpListener::bind` + `from_std`, which is what lets `cli::apply_reload` stay synchronous. Three traps: a role with **no socket parks** (that is `admin.enabled = false`, at startup as on a reload) rather than ending, because ending would close `incoming` and `accept` reads that as the accept task having *died*; the command channel closing is likewise **not** an ending — `Reloads::none()` drops every handle the moment the supervisor sees it will never fire, and a listener that stopped serving then would break every caller that serves no reloads; and the accept arm borrows the current socket while the command arm replaces it, so one pass produces a `Next` value rather than acting inside the `select!`. This grew out of `tls.rs`'s `TlsListener`, which already accepted TCP itself and already read its settings per connection — the whole reason a renewed certificate never needed a restart.
- `src/metrics.rs` — the Prometheus registry and the text exposition format, plus `split_matched_path`. Hand-rolled and dependency-free: the format is a `write!` per series, and every façade crate ships a **global recorder**, which this tree refused for `CryptoProvider::install_default` on the same grounds. `BTreeMap` behind a `Mutex` so `render()` is deterministic and a test can assert on it as text; label values are escaped (three escapes, and a collector rejects the *whole* scrape when one is missing). `Metrics::record_audit` takes an `AuditRecord`, so the certificate counters and the audit trail are written from one value and cannot drift. The pool gauge is read from `sqlx` at scrape time rather than tracked in parallel. **Held by `Assembly`, never by a `Generation`** — see the `Assembly` note.
- `src/middlewares/metrics.rs` — counts one request per (profile, route, status). Labels come from `axum::extract::MatchedPath`, which is the route *pattern*; the URI would mint a series per order id, and a Prometheus series is memory in this process *and* in the scraper for as long as it is retained. Two facts worth not rediscovering: `Router::layer` applies per route (so `MatchedPath` really is populated here, where a pre-routing layer such as `access` would see `None`), and it applies to the **fallback** too, so an unmatched request is counted under `ROUTE_UNMATCHED` rather than escaping the count. Layered onto the ACME router even though the exposition is served on a *different* socket — this is the only router that sees an ACME request, and the registry both share is an `Arc`. Added only when `metrics.enabled`, so an operator who has not asked for metrics pays neither the lock nor the allocation.
- `src/handlers/metrics.rs` — `get_metrics` and `MetricsState`, a newtype rather than `AppState` (which holds exactly one `Profile`, where the registry is the process's).
- `src/middlewares/access.rs` — request correlation *and* the access line, in one server-wide layer (was `request_id.rs` plus a per-profile `TraceLayer`). Reads or generates `x-request-id`, opens the `request` span every other line of that request nests under, and emits one `request_completed` carrying `status`/`latency_ms`. Three things the split got wrong and this fixes: `Span::current().record("request_id", …)` ran before `TraceLayer` had created a span and wrote to nothing; the root routes (`/health`, `/`, the http-01 responder) and the admission layer sat outside the profile routers, so `request_shed`/`request_deadline_exceeded` carried no id; and `DefaultOnResponse` emits under the `tower_http` target, which the shipped filter set to `warn`, so the access line was configured at INFO and silenced. Hand-written rather than `TraceLayer` because `on_response` never sees the request and so cannot drop `/health` to `debug`. The span declares `profile` as `field::Empty` and `build_router`'s outermost layer records it — the deferred-record pattern `verify_jws` also uses for `alg`/`account_id`. **`client_ip` is declared the same way but seeded here** with the `ConnectInfo` peer (canonicalized through `filter::canonical`, so a v4-mapped v6 is not spelled two ways across the two writers) and overwritten by `middlewares::filter` with the `ProxyPolicy`-resolved address. Seeding is not redundant with the overwrite: `/health`, the http-01 responder and every admission refusal sit outside all profile routers, so without it exactly the routes that carry no `profile` would also name nobody.
- `src/sqlite/status.rs` — `OrderStatus`/`AuthzStatus`/`ChallengeStatus`, generated from one table, plus `UnknownStatus` and `from_column`. Rust-side only: each variant's `as_str` is the byte-identical string the column already holds, so the frozen migrations and their `CHECK` constraints are untouched. It replaced ~30 comparisons against string literals spread over `handlers/`, `sqlite/` and the relay flow, where a typo compiled and silently changed policy.
- `src/templating.rs` — the `minijinja` loader both environments share (`notify` for `.j2` messages, `webadmin::pages` for `.html` pages). Sharing it cannot weaken the escaping rule: minijinja picks auto-escaping off the template *name*, never the loader.
- `src/sqlite/db.rs` — connection + automatic migration runner. `nonce.rs`, `account.rs`, `order.rs`, `authz.rs`, `eab.rs` — the models described under Architecture; DB methods return `Result<_, sqlx::Error>`.
- `src/sqlite/audit.rs` — `AuditEntry`/`AuditQuery`: append, read back, page, purge by age. No setter, no `UPDATE`. `event`/`actor_kind` come back as the **strings** they were stored as rather than as enums, so an older build reading a newer database renders an unrecognised event instead of refusing to load; `AuditEntry::event()` parses for callers that want the enum.
- `src/sqlite/upstream_order.rs` — maps a local order to the order the `relay` backend opened for it upstream. `order_id` as the primary key doubles as the concurrency guard: two finalize requests racing on one order can't both open an upstream order, since the second insert conflicts.
- `src/cli/audit.rs` (`audit list|show|cleanup`), `src/cli/filter.rs` (`filter show|explain`, argument marshalling only — `resolve_profile` lives in `src/cli/mod.rs`, hoisted when this became its second caller) and `src/cli/webadmin.rs` (`admin user`/`admin session`/`admin user totp`) are the three clap subtrees kept out of `mod.rs`.
- `src/admin/` — the operation layer **both front ends dispatch to and neither owns** (`src/cli/` and `src/webadmin/`): `prompt.rs` (`confirm`, over an injectable `BufRead`), `ops.rs`, `render.rs` (`render_*_json` **only** — the human line and detail renderings are the CLI's alone and live in `src/cli/render.rs`, which is where colour is woven in and therefore cannot reach a `--json` shape; order rendering additionally surfaces `revokedAt`/`revocationReason`, admin-only since `Order::to_json` carries no such fields — plus `certificatePem`, which `render_order_detail_json` adds and `render_order_json` deliberately does **not**, since the latter also renders every row of every listing and a page of fifty orders would carry fifty chains. The ACME `certificate` member beside it is the *URL*, unreachable from a browser), `password.rs`, `users.rs`, and the second factor: `totp.rs` (RFC 6238 over RFC 4226 on `ring::hmac`, plus the base32 *encoder* — nothing ever decodes it, so there is deliberately no decoder), `recovery.rs` (generation and normalisation), `mfa.rs` (where those two meet a database).
  - `revoke_order` takes an `Actor` and a `ClientContext` from its caller rather than deriving them: this layer is front-end agnostic, so the CLI supplies `Actor::cli` with an empty context (there was no request, and the row says so) and the web admin supplies `Actor::admin(username)` with the operator's own address — **the operator, not the certificate's owner**, since an administrative revocation attributed to the client would say the opposite of what happened. Its four non-revoking outcomes write no audit row: the operator is being told the state of things, unlike `POST /revokeCert`'s refusals, which are a remote party being turned away.
  - **Each destructive op comes in two forms**: a bare `delete_account(id, db)` / `delete_order` / `cleanup_nonces`, and a `confirm_*` wrapper taking `assume_yes` + `reader`. The CLI calls the wrapper; the web calls the bare one. Before the split an HTTP caller had to pass `true` and an empty reader, asserting a confirmation that never happened — and the `impl BufRead` generic made the function non-object-safe for no benefit on that path. The bare deletes return `Option<Deleted { cascaded }>`, the count being already computed for the prompt. `revoke_order` was never confirm-gated.
  - `password.rs` — PBKDF2-HMAC-SHA256 via `ring::pbkdf2`, 600 000 iterations, stored `pbkdf2-sha256$<iters>$<salt>$<hash>` (self-describing, so raising the cost or swapping to Argon2id is one branch plus rehash-on-login, never a migration). Deliberately **not** `argon2`: four crates into a CA's dependency graph — all re-audited by `cargo deny` at `all-features = true` — for a subsystem that is off by default and whose password is the bootstrap credential in a design that ends in a second factor. ~85 ms per hash in release, ~1.3 s in debug, which is why only two tests use the real constants and the rest go through the private `hash_with_iterations`.
  - `users.rs` — `create_user`/`set_password`/`set_status`/`delete_user`/`revoke_sessions`/`authenticate`. `authenticate` runs the KDF **even for an unknown username** (against `password::dummy_hash`), or login latency enumerates the operator table. `dummy_hash` is a `LazyLock` that **encodes** a fixed digest rather than calling `hash_password` — deriving one there costs a round the caller's `verify_password` then pays again, which made the unknown branch cost *twice* a known one and inverted the oracle instead of closing it; the test that catches this is `assert_eq!(dummy_hash(), dummy_hash())`, since the old spelling salted randomly. It and returns a four-variant `AuthOutcome` the caller collapses into one `invalid_credentials` *for the client* while logging which actually happened.
- `src/webadmin/` — the HTTP front end. `mod.rs` (`AdminState`, `build_admin_app`, `check_config` — `AdminState` holds the **same** `Arc<Auditor>` the ACME listener does, so an operator revoking through the panel writes into the one trail), `error.rs` (`AdminError`, `application/json`, stable snake_case codes — **never** `Problem`, whose every `typ` is a hardcoded ACME URN), `session.rs` (token mint/hash, the `__Host-` cookie, **five** extractors, the CSRF + origin gates, `LoginLimiter`, `AdminClientIp`, `PENDING_MFA_TTL`), `handlers/` (one file per resource, flat re-exports, plus `paging.rs`).
  - Routing: `GET /` redirects to `/ui/`, and the router's **fallback is HTML** — `/api`'s JSON fallback is scoped inside its own nest, so a script still gets the admin error shape on the paths a script uses. A strict `Content-Security-Policy` (`default-src 'none'`, no `unsafe-inline`/`unsafe-eval`) joins the five existing header layers.
  - **CSRF enforcement is structural, not a layer**: `AuthenticatedWrite` is the only way a mutating handler can reach a session, and constructing it runs the check — the reasoning that hoisted the media-type/`crit`/`url`/nonce checks into `AcmeRequest`. The residual risk (a new handler taking `Authenticated` by mistake) is closed by the table-driven suite in `tests/admin_api.rs`; **an endpoint missing from `mutating_endpoints()` is a review catch**.
  - **The two MFA-step routes are the one exemption**: `POST /api/session/mfa` and `POST /ui/login/mfa` run the origin gate and **no CSRF check**, because the challenge page is a plain form (sign-in must work with JavaScript off, which `tests/admin_pages.rs` pins for `login.html`) and `check_csrf` reads a header a form cannot set. `POST /ui/login` already makes exactly this trade one step earlier. Teaching `check_csrf` to also read a form field would be a second CSRF path, which is what `pages/auth.rs`'s shape exists to prevent. They live in their own `mfa_step_endpoints()` table with their own tests, so the exclusion reads as a decision rather than a gap.
  - **Three extractors carry the second factor**: `PendingMfa` (a live session that is *not* active — `resolve_session`'s mirror image, over a shared `resolve_live` that judges liveness and never `state`), `PendingMfaSubmit` (that plus the origin gate), and `EnrolWrite` (an `active` session, **or** a `pending_mfa` one **only when `!user.has_totp()`**). That last condition is the whole security of the type: a session that owes a *code* must never reach an enrolment route, or the factor is bypassable by enrolling a new one over it.
  - **Sign-in is two steps once a factor exists.** The password mints a short-lived `pending_mfa` session (`PENDING_MFA_TTL`, 5 min, a constant not a config key) reaching only the routes that finish the login, and proving a code **rotates** it into a fresh `active` one — a new token and a new CSRF token, never an `UPDATE state`, since the pending token crossed the wire before authentication completed. `record_success`/`mark_logged_in`/`log_login(true)` all moved to that promotion: leaving `record_success` at the password step would let somebody holding a correct password clear their own limiter bucket on every attempt and brute-force six digits. There are **two** promotion paths sharing `finish_mfa`/`finish_enrolment` for exactly that reason — the second is the `require_mfa` bootstrap, where setting a factor up *is* the second step, and before it was extracted the two front ends had already drifted (the API side doing none of the three, the pages side one).
  - **Guessing is bounded twice**: `LoginLimiter` by peer address, and `admin_sessions.mfa_attempts` by session — the latter because a `pending_mfa` cookie is valid from any address on purpose (`created_ip` is forensics, never compared), so the address-keyed bound alone gives an attacker `login_max_attempts` fresh guesses per source, of which one IPv6 /64 supplies 2^64. It is a single `UPDATE … RETURNING`, which is also what makes it race-free on a listener that deliberately has no admission control, and past the cap the pending row is **deleted** rather than merely refused.
  - **Changing a live factor takes the password again** (`check_step_up`, on `begin_totp`/`disable_totp`/recovery-code reissue, and only when a factor already exists): each of those also revokes every *other* session and supersedes the recovery codes, so without it one stolen cookie converts into a lockout only `admin user totp reset` on the host undoes. A first enrolment is ungated — it protects nothing, and a password there would stand in front of the `require_mfa` bootstrap. It runs `LoginLimiter` **before** the KDF and against the *same* bucket sign-in uses (a second budget would double the guesses against one password), and deliberately never `record_success` — clearing the bucket on a correct password is exactly what `sign_in` moved past the second factor to prevent. A corrupt stored hash records no failure, `decode` having failed before any work was spent. Still unbounded per *session*, so an attacker rotating addresses gets `login_max_attempts` each; closing that needs a column on `admin_sessions` and was judged not worth a migration for a secret behind 85 ms of PBKDF2.
  - `SameSite=Strict` is set but is **not sufficient alone**: SameSite is scoped to the registrable domain, not the origin — *different ports of the same host are same-site*, which is exactly this deployment. Hence the per-session token, plus an origin gate that also covers login (which carries no session yet).
  - `AdminState` is not `AppState`: the latter holds exactly one `Profile`, and this listener is cross-profile (revoking an order resolves **that order's own** profile's signer, or answers `409 profile_not_mounted`). `build_admin_app` takes `&[Arc<Profile>]` as a **slice** so it must be called before `build_app` consumes the `Vec` — the ordering is stated in the signature rather than rediscovered as a borrow error.
  - **No `#[instrument]` anywhere in this module** — a rule, not a preference: the access middleware already opens the request span, and the attribute tanks reported coverage (see the Coverage gotcha).
- `src/webadmin/pages/` — the HTML front end at `/ui`, the sibling of `handlers/` and built the same way (one file per resource, doc comment starting with the literal route, `not_found` per file, no `#[instrument]`). `mod.rs` (`pages_router`, plus `chrome`/`respond`/`respond_fragment`/`flash`/`flash_error`/`pager`), `templates.rs` (`EMBEDDED_TEMPLATES` + `build_environment` + `render`, a deliberate copy of `notify::build_environment`), `error.rs` (`PageError`, `redirect`, `escape_html`), `auth.rs` (`PageSession`/`PageSessionWrite`/`PageMfaPending`/`PageMfaSubmit`/`PageEnrolWrite`/`is_htmx`), `assets.rs`, `{session,account,accounts,orders,eab,misc}.rs`. Templates in `src/webadmin/templates/*.html`, assets in `src/webadmin/static/`. `orders::download_chain` (`GET /ui/orders/{id}/chain.pem`) is the one page route that serves neither a document nor a fragment: it is a `GET`, so its absence from `mutating_page_endpoints()` is by right rather than by omission, and it takes `PageSession` — a certificate is public once issued, but *which orders exist* is not. Its `Content-Disposition` filename comes from the **stored** id, not the path-supplied one, since that string is interpolated into a header. `mfa/` and `account/` hold the second factor's five: `mfa/challenge.html` is standalone like `login.html` and branches on `step`; `mfa/_setup.html` is included from both the sign-in flow (a plain form) and the account page (htmx), so it renders no form element of its own; `account/_mfa.html` is the swap target and `account/_card.html` the card inside it, split so `account/_codes.html` can show a fresh set *and* the up-to-date card without nesting two elements carrying one `id`.
  - **`.html`, never `.j2`, is a security control**: minijinja picks auto-escaping off the template *name*, and the notify templates are `.j2` precisely so escaping is off for them. A page template renamed `.j2` turns an account contact or an EAB label into stored XSS. `auto_escaping_is_on_for_pages_and_off_for_notify` pins both directions.
  - **`PageError` is separate from `AdminError`** because the latter's `IntoResponse` is JSON-only and a browser needs to *arrive* at `/ui/login`. Its shape is fixed at construction, not in `into_response`: a `303` cannot serve htmx, since `fetch` follows the redirect before htmx sees a header and the sign-in page would be swapped into the clicked element — so the extractors read `HX-Request` from `parts.headers` and pick `303 + Location` or `204 + HX-Redirect`. It renders **no template** (a `const` document plus a local `escape_html`), the reasoning that keeps `AdminError`'s body a `json!` literal.
  - **`PageSessionWrite` wraps `AuthenticatedWrite`**, so the origin/session/CSRF gates stay one implementation. The token reaches the browser as `hx-headers` on `<body>` in `layout.html` and returns as `X-CSRF-Token`, which is why `check_csrf` needed no second path and why htmx was chosen over plain forms. `tests/admin_pages.rs::mutating_page_endpoints()` is the `/ui` twin of the API's table — **a `/ui` route added and not added there is a review catch**.
  - Two htmx settings in `layout.html` are load-bearing and documented in `src/webadmin/static/README.md`: `includeIndicatorStyles: false` (htmx would otherwise inject an inline `<style>` the CSP blocks; the rules live in `admin.css`) and `responseHandling` (htmx does not swap non-2xx by default, so a `409` would fail silently instead of showing a banner). The refusal split is "the row's state is a banner, the server's problem is a page".
  - `htmx.min.js` is vendored (2.0.10, `0BSD`). `cargo deny` audits crates and cannot see it, so `src/webadmin/static/README.md` — version, URL, SHA-256, licence, refresh command — is the only provenance record.
- `src/webadmin/handlers/audit.rs`, `src/webadmin/pages/audit.rs` — the JSON and HTML audit surfaces, **read-only by design**, which is why neither appears in `mutating_endpoints()`/`mutating_page_endpoints()`: there is no route to list.
- `src/sqlite/admin_user.rs`, `admin_session.rs`, `admin_recovery_code.rs` — the three admin models. The session layer **never sees the plaintext token**; it arrives already hashed from `webadmin::session`, which is also why there is no `find_by_id` (the hash *is* the lookup key).

## Inline unit tests

- `src/extractors/signature.rs` — JWS verification (EC/RSA, rejection paths), the DER-SPKI round-trip, `jwk_thumbprint` against RFC 7638's worked example, every `SignatureError`'s `Display`, and crafted DER driving each `spki_parts`/`spki_to_jwk` encoding error (a corrupted `accounts.pubkey` row must refuse the request, never authenticate somebody else). **`acme.rs` has no inline tests** — every one named here is in `signature.rs`, and the extractor itself is covered only from outside (`tests/jws_rejections.rs` is the table-driven suite, `tests/db_failures.rs` the failure paths). Worth knowing before assuming a change there is unit-tested: it is the core of the codebase and nothing in this file exercises it.
- `src/sqlite/nonce.rs` — single-use, expiry, cleanup, TTL edges.
- `src/sqlite/account.rs` — `find_or_create` dedup, contact/deactivate persistence.
- `src/sqlite/order.rs` — create/find round-trips, `to_json` shapes, finalize/mark_invalid/revoke persistence, `find_by_cert_serial`, `with_client` persisting while `to_json` exposes neither column.
- `src/audit/*` — every `AuditEvent` round-tripping through its stored form and knowing its own outcome (the two `CHECK`s are written against exactly these strings), the four `Actor` constructors, the `User-Agent` cap, the empty-header rule, `from_config`'s two shapes, and every way a PTR lookup ends (a record, several collapsing to the first, none, a resolver failure, a timeout, `reverse_dns` off). Plus the one that matters most: a failed write is **swallowed**, because a certificate already signed must not become a 500 the client retries into a second issuance.
- `src/sqlite/audit.rs` — round trip, the derived `outcome`, every filter narrowing page *and* count together, paging with the `id` tiebreak (every row lands in the same whole second), `cleanup`'s strict `<` cutoff, `to_json` omitting rather than nulling, an unknown event string still loading, and **a row surviving the account and order it names being deleted**.
- `src/sqlite/job.rs` — every guarded statement: the partial identity index (a second *live* enqueue refused, one after `done` not), the claim taking the oldest eligible row and only once, `run_at`/kind filtering, and **every settlement refusing to write when `lease_owner` differs**. Plus `retry` keeping the attempt count where `reschedule` resets it, `reclaim_expired` leaving `attempts` alone, `release_owned` scoped to one runner, `cleanup`'s strict `<` cutoff over terminal rows only.
- `src/jobs/*` — the outcome table driven by a scripted handler with a call counter, the only thing that can prove `abandon` fired **exactly once** on exhaustion, on `Failed` and on a passed deadline, and never on an intermediate `Retry`. Plus a retry landing past the deadline retiring rather than waiting, a panicking handler leaving the job retryable rather than wedging its lease, a handler exceeding its lease timing out as a retry, an unregistered kind never claimed, the `Notify` path running a job well inside a 60-second `poll_interval_ms`, shutdown releasing leases, `recover` running once and safely repeatable, the registry refusing a duplicate kind, and `backoff`'s table incl. jitter **never exceeding** `retry_max_seconds`. Then the four for `[jobs]` reloading: `Permits::resize` growing whole and shrinking only as slots come back, a lowered `poll_interval_ms` landing without waiting out the old one (the reason the config cell has a `select!` arm at all), a raised `max_concurrent` reaching a pool that is **already saturated** — the only state anybody raises it from, and the one the old parking `drain_ready` could observe nothing in — and a runner over `spawn_runner`'s dropped-sender cells staying idle, which counts passes via an always-retrying handler with zero backoff, since a spin is invisible to every other assertion here. `mod.rs` adds the queue half: a reloaded `max_attempts` reaching a clone taken *before* the change while a row queued before it keeps its own budget.
- `src/jobs/sweep.rs` — each target deleting from its own table and answering a distinct `kind`, the three intervals keeping their floors (a `ttl / 4` of zero would be a `tokio::time::interval` busy loop taking the WAL writer lock), `recover` queueing one row however often it runs, and the law the module rests on: **no target ever returns `Failed`**, driven against a closed pool.
- `src/admin/render.rs` — `render_account_json` omitting rather than nulling the five traceability members (a template's `{% if account.createdIp %}` has to be asking what it looks like it is asking), `render_order_json`'s admin-only members, and every `_json` shape carrying no secret.
- `src/cli/render.rs` — the three shapes of client (address + name, address alone, none) `render_client` renders for **both** `render_audit_line` and `render_account_line`, and `render_audit_detail_text`/`render_account_detail_text` omitting every absent field. Every one of those runs against `Palette::plain()` and asserts the exact bytes — several on an exact padding run or with `ends_with` — which is what pins that colour did not leak into the default path. Each renderer then gets a coloured case asserting `strip_ansi(painted) == plain`, the regression for wrapping a field *before* padding it. Note `render_order_line`'s `{:<9}` has never padded anything (`OrderStatus`'s `Display` is a bare `write_str`, which ignores the width) and is deliberately left ragged: the contract is identical bytes with the palette off, not the layout the format string looks like it asks for.
- `src/cli/style.rs` — the `resolve` precedence table (every choice × TTY × `NO_COLOR` set/empty/unset, including `always` beating both vetoes), the status vocabulary by meaning rather than by source table, an unrecognised word left **unpainted**, and the invariant every plain-output assertion in the crate rests on: `Palette::plain()` returns its argument unchanged.
- `src/cli/audit.rs` — the `--event`/`--outcome` refusal by name, both output shapes, the window clamps, the confirm gate on `cleanup`.
- `src/cert.rs` — serial/SPKI extraction, leaf-from-chain, revocation-reason validation.
- `src/sqlite/authz.rs` — Authorization/Challenge CRUD, mark_valid/invalid, per-type tokens, the wildcard storage convention.
- `src/signer/mod.rs` — the reload pass over `build_backends`: an unchanged configuration handing back the **same `Arc`** (the only assertion that can tell reuse from a rebuild that adopted everything), an edited one rebuilt over the running ledger and asserted through the **CRL in both directions** (a one-way copy passes the first half), a changed `Egress::identity` rebuilding a backend whose `[signer]` never moved, a profile mounted and unmounted, and a rebuild over a *different* `crl_path` adopting nothing.
- `src/signer/local_ca/mod.rs` — issuance (chain shape, identifier/SAN mismatches → `BadCsr`), a `CA:TRUE` CSR still yielding a non-CA leaf, distinct serials per issuance, the RFC 5280 §5.3.1 reason-code table (7/11/out-of-range → "no reason recorded", never a refusal to revoke), CRL lifecycle (empty → revoked, idempotent revoke, ledger surviving a reload through the sidecar *and* through an adopted `Arc` — the latter asserted both ways, since the shared cell is what covers the window the file cannot), the prune (expired dropped, live kept, the `CLOCK_SKEW_ALLOWANCE` grace granted then expiring, an entry with **no** recorded expiry never dropped however old, `revoke` recording the leaf's `notAfter` at all, and a prune with nothing to drop leaving the CRL byte-identical), the `crl_number` rising across a restart, across a prune that *shortens* the list — the regression the whole envelope exists for — and above whatever the 0.1.0 bare array could have published, plus the two the sweep needs: one handler registering cleanly over **two** CAs (the duplicate-`kind` trap, invisible to any single-profile test) and a prune whose persist fails still answering `Reschedule`, asserted after proving the persist really failed, and the two pointer extensions driven through `load_or_generate` rather than a policy handed to a constructor — only that path proves config → `LeafPolicy` → `assemble` → `issue` is wired — plus the regression that an unconfigured CA writes neither.
- `src/signer/local_ca/policy.rs` — the DER an operator never sees: two `authorityInfoAccess` vectors, one short-form and one whose outer SEQUENCE passes 127 bytes (the long-form length branch a hand-rolled encoder gets wrong), every URL refusal asserted on its message, the empty list emitting no extension rather than an empty SEQUENCE.
- `src/signer/local_ca/key.rs` — `KeySource::parse`'s two values and its error, `read_pin`'s `pin_file`-beats-`pin` precedence and its trailing-newline trim (an untrimmed PIN is a *wrong* PIN, which costs a login attempt), and the `Issuer<'static, KeyPair>` → `Issuer<'static, CaSigningKey>` regression: a leaf signed through the enum must verify against the CA.
- `src/signer/local_ca/pkcs11.rs` (`#[cfg(feature = "hsm")]`) — the pure conversions that silently corrupt certificates when wrong (raw `r‖s` → DER with a high-bit half staying positive and leading zeros dropped; `CKA_EC_POINT` unwrapped from its `OCTET STRING`, short and long form, and taken bare when a token sends it that way), plus a `softhsm` submodule driving a **real SoftHSM2 token** end to end: it creates the token and generates the P-256 key through `cryptoki` itself (so `opensc`/`pkcs11-tool` is not a prerequisite, only the module `.so`), self-signs the CA *through the token*, then asserts an issued leaf and the CRL both verify against it. Skips rather than fails when no module is found.
- `src/signer/custom.rs` — the four hooks' env/stdin/exit-code contracts (`issue`'s reserved `BadCsr` exit code, `revoke`'s idempotency-is-the-script's-job contract, `crl`/`renewal_info` gated by their `supports_*` flags and never spawning when off), environment isolation, timeout + kill-on-drop.
- `src/signer/relay/dns01.rs` — the RFC 2136 exchange against a loopback nameserver holding one UDP socket and one TCP listener on the same port: NOERROR, a REFUSED rcode, a mismatched response id, an undecodable answer, and a truncated UDP answer forcing the TCP retry (a TSIG-signed update readily exceeds 512 bytes, so that fallback is a normal path).
- `src/tls.rs` — `from_config` (disabled/generate/reload/mismatch), on both listeners' halves.
- `src/listener.rs` — the loopback suite, over a real socket and a real `axum::serve`: a **replaced socket** serving the new port with the old one *released* (not merely ignored), a `tls.enabled` flip both ways on one unchanged port, `Close` refusing connections and a later `Serve` reopening, a swapped certificate reaching the next connection, `ConnectInfo` surviving both `MaybeTls` variants (the cleartext one is new — that path used to be `axum::serve`'s own accept loop), a failed and a stalled handshake, and `bind_blocking`'s two answers, since a reload quotes its error rather than panicking.
- `src/pemfile.rs` — chain/key parsing, label mapping, file permissions.
- `src/challenge/*` — `from_config` validation, dispatch, per-validator behavior (`http_01` redirects/timeouts via a stub plus a loopback `HyperFetcher` test; `dns_01` via a stub resolver; `tls_alpn_01`'s `verify_acme_identifier` plus a loopback handshake proving the critical extension survives `rustls-webpki`). Both validators' `Debug` impls are asserted on — they render the configured policy, since neither `dyn HttpFetcher` nor `dyn TlsAlpnProbe` is `Debug`.
- `src/error.rs` — every `Problem` constructor's status/body.
- `src/config/` — defaults via `Config::load()`, `config.toml.example` parsing and matching, `LIST_KEYS` round-tripping.
- `src/sqlite/db.rs` — connect/migrate, WAL + `foreign_keys`, CASCADE, CHECK constraints.
- `src/filter/*` — the Kleene truth table in full (21 rows, table-driven — the whole design rests on it), short-circuiting proven with a call-counting stub (a skipped operand and a passing one produce the same verdict, so the counter is the only thing that can prove a skip) and the converse that an `Undecided` left operand *does* evaluate the right, memoisation, first-match-wins, warn mode, the empty-applicable-set law, all three detail-selection branches; every `build.rs` startup refusal; the glob vocabulary; `explain`'s renderings incl. a skipped check rendered as *skipped*; list/CIDR/regex helpers, `ProxyPolicy::resolve`, per-check behavior (`ip_allow`/`path`/`reverse_dns`/`identifiers`/`eab`); and `ipam`'s policy half against a stub `Ipam` — the dns/ip/cn/other rules, `Unknown` → `Denied` worded differently from an unlisted name, `IpamError` → `Internal`, the registry's budget surfacing as an ordinary `Internal`, a CN-only CSR asking the inventory **nothing** (a call counter proves it), and `from_config`'s three name arms (`ipam` builds, `ipam` with no backend is a startup error, `netbox` is refused by name).
- `src/ipam/*` — `from_config`'s four arms and the registry's timeout; the `sources` vocabulary (empty, unknown, a source belonging to the other backend, each refused by name); `AddressNames::Unknown` not being an empty `Known`; `normalize`/`field_values`/`value_kind`; the shared transport (`http.rs`: body cap, the excerpt on a non-2xx, an unreadable body, a refused connection, and **the status surviving on the error** so phpIPAM can read a 404 as an answer). Each backend gets a stub-driven policy suite plus a loopback wire suite. NetBox: every source proven to make **zero** queries when left out (`AtomicUsize` counters), the two service-address sources proven unions rather than fallbacks, an off-role object dropped client-side, and — the one that matters — **an interface in no FHRP group contributing nothing, with the group-addresses query never made**; the wire half pins the repeated `role=`/`fhrpgroup_id=` parameters (comma-joining them would match nothing) and `interface_id` taken from `assigned_object.id` while the device id comes from the object nested under it. phpIPAM: the comma split, the `token` header rather than `Authorization`, `404` → `Unknown`, and the id shim (integers rendered as strings in some versions; `0`/`"0"`/`null` all mean "no device").
- `src/proxy.rs` — table-driven `no_proxy` (exact, `.example.com`, the bare domain doing the same, `notexample.com` **not** matching, `*`, a literal, a CIDR, a hostname never matching a CIDR rule), the unconditional loopback/`localhost` bypass, `http_url` set without `https_url` leaving https targets direct, a fixed `Basic` vector plus a percent-encoded credential, neither `Debug` nor `redacted()` carrying the password, every refusal asserted on its message, and the environment fallback under `ENV_LOCK` (config beats env, `HTTPS_PROXY` honoured, `HTTP_PROXY` ignored, empty string is unset).
- `src/http_client.rs` — `request_target` in both forms, `connect_authority` keeping a default port, then a `mod loopback` driving a **fake CONNECT proxy** (`testutil::FakeProxy` with an `AtomicUsize` connection counter): a tunnel carrying bytes end to end, the squid-shaped `HTTP/1.0 200 Connection established` + `Proxy-Agent` reply still tunnelling, absolute-form forwarding with `Proxy-Authorization`, a `407` surfacing status and body excerpt with the password redacted, a dead proxy naming the *proxy* rather than the origin, a `no_proxy` hit leaving the counter at **zero** (the only assertion that can prove a bypass), and https-through-a-tunnel end to end against a loopback rustls listener — proving the double `TokioIo` wrap, SNI carrying the **origin's** name, and the proxy credential stopping at the proxy.
- `src/dns.rs` — TXT character-string concatenation, lossy non-UTF-8 decoding, plus a loopback nameserver driving the real `HickoryResolver` (the PTR root-strip, and an empty NOERROR answer reading as "no records" rather than a failed lookup). Reverse lookups use a documentation-range address, not `127.0.0.1`: hickory answers loopback from its own built-in zone without reaching a nameserver.
- `src/notify/*` — every `NotifyEvent` variant answered by every accessor (a variant added to only some `match`es would surface as a silently empty field) **and round-tripping through its own JSON**, since a queued delivery is that JSON and nothing else; `payload_is_tagged_with_its_own_hook` pins the exact object a `custom` script reads on stdin. Plus `from_config`'s per-backend arms and their validation failures, two `custom` entries getting **distinct slot ids** (two `webhook` entries likewise), `dispatch` writing one row per *wanting* backend and **none** for a filtered-out one, a database failure swallowed there, and each backend's delivery path against a loopback listener (`webhook` — which also pins the configured verb reaching the wire, an operator header overriding a default, and the `tojson` regression: a message holding a quote and a newline still parses as JSON) or a dead port (`email`). The retryable/permanent split is asserted where it is decided: a missing and an uncompilable template, and `retryable_status`'s table.
- `src/notify/job.rs` — the outcome mapping (delivered → `Done`, retryable → `Retry`, permanent → `Failed`) and the three things that retire a delivery without reaching a backend (unknown profile, unknown backend id, four shapes of unreadable payload). Plus `abandon` surviving the unparsable payload that retired the job — an arm reachable precisely because `run` returns `Failed` for it — and `recover` queueing nothing, since a queued row is already durable.
- `src/reload.rs` — the `FROZEN` table driven one key at a time (one entry now), each refusal asserted to name **its own** key (a projection reading the wrong field would answer `Ok` or name another), plus a set-equality check between the cases and the table so neither can drift; every `[logging]` key reloading where the whole section used to be refused, and all seven of both the `[jobs]` keys and the **listener** keys, plus the profile set, both `[signer]` sections and `[dns]`/`[proxy]` — that last one belongs beside the freeze rather than only in `tests/reload.rs`, because a key left in the table would make the whole rebinding path — and the whole carried-state path — unreachable, the refusal happening before anything is built; an unchanged configuration not being refused (without which every assertion above passes while production refuses everything); the channel coalescing a signal storm; and the three router-cell properties — a request after a swap reaching the new router, the wrapper never answering in place of the inner fallback, and `HEAD` keeping its `Content-Length` through the doubled `top_level` nesting.
- `src/cli/*` — every command arm's failure against an in-memory database: unknown ids, `order revoke`'s outcome mapping and its profile/signer resolution, `upstream`'s stdin-only secret handling, `admin user`/`admin session`'s arms (duplicate username, policy failure, `--password-file`, delete cancelled vs confirmed), plus the negative test asserting `--password` does not parse and its positive twin asserting `--version` reports the crate version (clap generates that flag only because `#[command(version = …)]` says so, and the bug report template tells people to run it). `logging.rs` additionally proves the reload seam: publishing with no subscriber installed is a `false` rather than a panic, a swapped filter moves `LevelFilter::current()` (the only assertion that can prove the interest cache was rebuilt rather than a new layer merely stored), and a format swap keeps the filter's level hint — the regression for the `and_then` argument order. Those install a *global* subscriber, which only nextest's process-per-test makes possible. `serve_on` is driven end to end on an ephemeral port (cleartext and TLS, a real request, a graceful shutdown) and `serve_on_with` on **two** ephemeral ports: `/health` on both, the admin API answering `401` on one and `404` on the other (converse for `/directory`), then **one** shutdown trigger stopping both — the regression test for the `watch` split, before which `shutdown` was an `impl Future` consumed once and only one listener stopped.
- `src/admin/password.rs` — hash/verify round trip, wrong password, every `PasswordError` branch (truncated field, unknown prefix, non-numeric/zero iterations, bad base64, short salt, truncated digest), `needs_rehash` both ways, the policy's character-not-byte counting, and the two costs coexisting in one self-describing format (`hash_generated_secret` at 10 000 iterations, plus the trap that `needs_rehash` reports `true` for every one of them).
- `src/admin/totp.rs` — **RFC 4226 Appendix D** (6-digit, counters 0–9) and **RFC 6238 Appendix B**'s SHA-1 rows (8-digit, which is why `hotp` takes `digits` at all), **RFC 4648 §10** base32 over every residue class of the 5-bit chunker, the ±1 skew window accepted and ±2 refused with the matched step reported, table-driven shape rejection before any HMAC runs, `step_at` not straddling the epoch, and the provisioning URI surviving a username holding `/`, `?`, `#` and a space as **one** path segment.
- `src/admin/recovery.rs` — ten distinct codes from the base32 alphabet in the `K7QF2-3BXTM` grouping, `normalize` absorbing case/separators/whitespace idempotently, and the shape check refusing `0`/`1`/`8`/`9` (the four characters the alphabet omits, which is why there is no confusable fixup).
- `src/admin/mfa.rs` — the four `MfaOutcome`s, TOTP tried before the PBKDF2 scan, a code accepted once and `Replayed` thereafter, a wrong code **not** advancing the guard, a recovery code spent once and retyped every way a human retypes one, regeneration superseding, `disable_totp` clearing all three columns *and* the codes, and a factor change revoking every other session.
- `src/admin/users.rs` — create/normalize/duplicate/policy, `set_password` and `disable` revoking sessions, `authenticate`'s four outcomes, the rehash-on-login path, and a corrupt stored hash refusing the login rather than erroring.
- `src/webadmin/error.rs` — every constructor's status, code and body; `Retry-After` only on the limiter; the three login failures byte-identical; a `sqlx` error logged and answered generically.
- `src/webadmin/session.rs` — token length/encoding, `hash_token` against a fixed vector, every cookie attribute, **table-driven cookie parsing** (quoted value, duplicate name in one header and across two, segment with no `=`, a name merely containing ours — the duplicate cases are the session-fixation vector), the CSRF compare incl. differing lengths, the origin gate, the limiter (under/at/over/rollover/pruning), the reaper. The MFA extractors are covered through the router (`tests/admin_api.rs`), including the mirror-image pair refusing each other's session and **`EnrolWrite` refusing a `pending_mfa` session whose user `has_totp()`** — the refusal the whole feature turns on.
- `src/webadmin/mod.rs` — `check_config`'s refusals, every loopback spelling, and an unparseable bind treated as non-loopback (conservative on purpose).
- `src/webadmin/handlers/paging.rs` — clamping, the negative offset, a `page_size_max` of zero still yielding a usable page.
- `src/webadmin/pages/templates.rs` — every embedded template compiles, a `template_dir` file beating the default (and only for that one file), and `auto_escaping_is_on_for_pages_and_off_for_notify`. Note minijinja escapes `/` too, so `</script>` renders as `&lt;&#x2f;script&gt;`.
- `src/webadmin/pages/error.rs` / `auth.rs` / `assets.rs` — both redirect spellings, the escaped error document, `escape_html`'s five delimiters, a `403 csrf_failed` **keeping its status** rather than bouncing to sign-in (the session is live; a redirect would explain nothing), `is_htmx`'s parsing, and the asset allowlist refusing anything outside its two names.
- `src/webadmin/pages/mod.rs` — `pager`: both ends of a result set, an empty one, and the regression that matters — **the filters survive a page step**, url-encoded, with an empty one absent rather than `status=`.
- `src/sqlite/admin_user.rs` / `admin_session.rs` — CRUD, the `UNIQUE(username)` conflict, the FK and both `CHECK`s, delete cascading to sessions, `cleanup` removing only expired/idle rows, `to_json` leaking neither hash nor CSRF token. Plus the second factor: the four `totp_*` setters persisting and syncing in memory, `confirm_totp` with nothing pending leaving a live factor alone (a double-submit must not clear one), **`claim_totp_step` refusing a step it has already seen** (the replay guard, in SQL), `create_pending`'s state and short deadline, `promote` yielding a different token *and* CSRF token while deleting the old row and returning `None` the second time, and `cleanup` sweeping an abandoned pending row with no special case.
- `src/sqlite/admin_recovery_code.rs` — `replace_all` superseding the whole previous set in one transaction, `consume` returning `false` the second time (the single-use guard, which is why this is a table), the cascade, `to_json` carrying no hash.