Expand description
§ctlog - canonical Certificate Transparency (crt.sh) client
Single source of truth for “query a Certificate Transparency log for every name ever certified under a domain, and turn the response into a clean, deduplicated host list” across the Santh fleet.
Before this crate existed, the crt.sh query+parse was reimplemented in at least five places with divergent behavior:
wafrift_recon- bounded chunked read, 30s timeout, typed errors, case-normalized, sorted+deduped. The most defensive copy.gossansubdomainctsource andoriginssl_certscanner - bounded read via their own helpers,*.stripped, lowercased.scaldsubdomain discovery - retry/backoff, URL expansion.karyxorigin-discoverycertificate- unboundedResponse::json(OOM on a hostile mirror), no wildcard in the query URL (so it only saw apex certs and missed every subdomain), and no case normalization (a mixed-case base domain leaked into results).
Folding them onto this crate removes the duplication and fixes karyx’s three latent bugs in one move. The crate is split into two layers so consumers that already own a transport stack don’t pay for one they won’t use:
- Pure layer (
crtsh_query_url,parse_crtsh_hostnames,parse_crtsh_subdomains,CrtShEntry) - JSON + a defensive URL encoder, no async/HTTP. Consumed by gossan (itsScanClient+ governor rate-limiter) and scald (its retry/backoff client), which feed an already-fetched body in. - Fetch layer ([
discover_subdomains_ct], [discover_subdomains_ct_with], behind thefetchfeature) - the chunk-bounded, timeout-guarded read wafrift hardened. Consumed by wafrift and karyx, which have no bespoke crt.sh transport.
§Normalization contract
parse_crtsh_hostnames / parse_crtsh_subdomains apply the union
of every consumer’s normalization, which is also the most complete:
- Split each
name_valueon\n(crt.sh packs the full SAN set of a cert into one newline-joined field). - Trim surrounding whitespace.
- Strip a leading wildcard label
*.so*.api.example.comstill yields the concreteapi.example.comhost - a real subdomain the drop-the-whole-entry copies (wafrift, scald, karyx) silently lost. - Lowercase (ASCII) so a mixed-case apex compares equal to the queried domain and doesn’t leak into a subdomain list.
- Drop empty strings and anything still containing
*. - Sort and deduplicate.
§Safe defaults
- Input size (parse): bounded by the caller - the parse functions
take an in-memory
&strthe caller already size-limited. The fetch layer enforces [CT_RESPONSE_MAX_BYTES] itself. - Outbound network: none in the pure layer; the fetch layer makes
exactly one GET to crt.sh under [
CT_QUERY_TIMEOUT]. - Process spawning / filesystem writes / credential exposure: none.
Structs§
- CrtSh
Entry - One row of a crt.sh
output=jsonresponse.
Enums§
- CtError
- Errors returned by the
ctlogparse and fetch layers.
Functions§
- crtsh_
apex_ query_ url - Build a crt.sh query URL that matches certificates for the apex
domain. - crtsh_
query_ url - Build the canonical crt.sh JSON query URL for
domain. - parse_
crtsh_ hostnames - Parse a crt.sh
output=jsonbody into the full set of normalized host names, including the queried apex. - parse_
crtsh_ subdomains - Parse a crt.sh
output=jsonbody into normalized subdomains, excluding the querieddomainitself.