Skip to main content

Crate ctlog

Crate ctlog 

Source
Expand description

§ctlog - canonical Certificate Transparency (crt.sh) client

Single source of truth for “query a Certificate Transparency log for every name ever certified under a domain, and turn the response into a clean, deduplicated host list” across the Santh fleet.

Before this crate existed, the crt.sh query+parse was reimplemented in at least five places with divergent behavior:

  • wafrift_recon - bounded chunked read, 30s timeout, typed errors, case-normalized, sorted+deduped. The most defensive copy.
  • gossan subdomain ct source and origin ssl_cert scanner - bounded read via their own helpers, *. stripped, lowercased.
  • scald subdomain discovery - retry/backoff, URL expansion.
  • karyx origin-discovery certificate - unbounded Response::json (OOM on a hostile mirror), no wildcard in the query URL (so it only saw apex certs and missed every subdomain), and no case normalization (a mixed-case base domain leaked into results).

Folding them onto this crate removes the duplication and fixes karyx’s three latent bugs in one move. The crate is split into two layers so consumers that already own a transport stack don’t pay for one they won’t use:

  • Pure layer (crtsh_query_url, parse_crtsh_hostnames, parse_crtsh_subdomains, CrtShEntry) - JSON + a defensive URL encoder, no async/HTTP. Consumed by gossan (its ScanClient + governor rate-limiter) and scald (its retry/backoff client), which feed an already-fetched body in.
  • Fetch layer ([discover_subdomains_ct], [discover_subdomains_ct_with], behind the fetch feature) - the chunk-bounded, timeout-guarded read wafrift hardened. Consumed by wafrift and karyx, which have no bespoke crt.sh transport.

§Normalization contract

parse_crtsh_hostnames / parse_crtsh_subdomains apply the union of every consumer’s normalization, which is also the most complete:

  1. Split each name_value on \n (crt.sh packs the full SAN set of a cert into one newline-joined field).
  2. Trim surrounding whitespace.
  3. Strip a leading wildcard label *. so *.api.example.com still yields the concrete api.example.com host - a real subdomain the drop-the-whole-entry copies (wafrift, scald, karyx) silently lost.
  4. Lowercase (ASCII) so a mixed-case apex compares equal to the queried domain and doesn’t leak into a subdomain list.
  5. Drop empty strings and anything still containing *.
  6. Sort and deduplicate.

§Safe defaults

  • Input size (parse): bounded by the caller - the parse functions take an in-memory &str the caller already size-limited. The fetch layer enforces [CT_RESPONSE_MAX_BYTES] itself.
  • Outbound network: none in the pure layer; the fetch layer makes exactly one GET to crt.sh under [CT_QUERY_TIMEOUT].
  • Process spawning / filesystem writes / credential exposure: none.

Structs§

CrtShEntry
One row of a crt.sh output=json response.

Enums§

CtError
Errors returned by the ctlog parse and fetch layers.

Functions§

crtsh_apex_query_url
Build a crt.sh query URL that matches certificates for the apex domain.
crtsh_query_url
Build the canonical crt.sh JSON query URL for domain.
parse_crtsh_hostnames
Parse a crt.sh output=json body into the full set of normalized host names, including the queried apex.
parse_crtsh_subdomains
Parse a crt.sh output=json body into normalized subdomains, excluding the queried domain itself.