certmagic
Automatic TLS certificate acquisition, renewal, and maintenance for Rust servers.
Overview
CertMagic manages TLS certificates for you: it obtains certificates from ACME CAs (Let's Encrypt, ZeroSSL, …), renews them before expiry, staples OCSP responses, coordinates with other instances through shared storage, and can even obtain certificates on-demand during the TLS handshake.
The crate automates the full certificate lifecycle and ships an idiomatic
Rust API (async traits, typed events, thiserror, Send + Sync everywhere).
Feature highlights
- ACME v2 client, self-developed: directory discovery, replay-nonce pool, account management with External Account Binding, order state machine, revocation, ARI (renewal information) with suggested-window jitter.
- Four challenge solvers: HTTP-01 (reference-counted shared listener +
framework-agnostic handler), TLS-ALPN-01 (RFC 8737 challenge certificates),
DNS-01 (pluggable
DnsProvider+ propagation checks), and distributed solving through shared storage — cluster-wide issuance without sticky sessions. HTTP-01 helpers validate unpadded base64url tokens, tolerate query strings and one router-added trailing slash, and expose an opt-in blind-solving fallback for deployments that cannot retain challenge state. - Handshake-time issuance and on-demand TLS: serve a certificate the first
time a name is seen, gated by your
DecisionFuncor explicit host allowlist; if neither gate is configured, issuance is denied. - Certificate cache and maintenance loop: SAN-indexed in-memory cache, renewal windows (fixed ratio or ARI-driven with jitter), OCSP stapling with a hand-rolled RFC 6960 codec (request building, delegated-responder authorization, ECDSA/RSA/Ed25519 signature verification).
- Storage abstraction with distributed locking: default file storage with atomic writes and heartbeat lockfiles; bring-your-own backend (Redis, etcd, S3, …) for multi-instance clusters.
- Separate certificate store: certificates and private keys can use a
dedicated
CertStorebackend while ACME accounts, locks and OCSP data stay on the ground-truthStorage. - Production hygiene: retry with hand-tuned 30-day backoff budget, per-name job deduplication, sliding-window rate limiter, typed event hooks, panic-proof maintenance loop.
- Operational hooks: cache lifecycle callbacks, configurable HTTPS redirect host policy, custom DNS resolvers for propagation checks, and optional suppression of automatic replacement after an OCSP revocation report.
Installation
[]
= "0.2"
= { = "1", = ["full"] }
The default build uses the AWS-LC-RS crypto provider and enables the ZeroSSL
issuers. RSA key generation requires the optional rsa feature. For a smaller, portable build using the
Ring provider, disable default features and select the required capabilities
explicitly:
[]
= { = "0.2", = false, = [
"ring", "file-storage", "http-01", "dns-01", "ocsp",
] }
The ring and aws-lc-rs provider features also select the matching
x509-parser signature-verification backend (verify and verify-aws,
respectively), so certificate verification uses the same backend as rustls,
rcgen, and reqwest. Select exactly one provider in a --no-default-features
build; optional runtime modules each have an independent feature flag.
The rsa feature is opt-in because it pulls in the RustCrypto rsa crate.
Deployments that only issue ECDSA or Ed25519 certificates should leave it off
to keep the dependency tree smaller and avoid RSA-specific timing-sidechannel
risk. RustSec RUSTSEC-2023-0071 remains explicitly tracked because the
upstream project has not published a fixed release yet.
Upgrading from 0.1 to 0.2
Version 0.2.0 adds optional Redis/etcd coordination and S3, Vault KV v2 and AWS Secrets Manager certificate stores. See the 0.2.0 changelog and the backend sections below for configuration and validation limits.
- FileStorage coordination: stop all cooperating instances before upgrading,
then restart them together. The new lock protocol requires OS file locks on
the shared filesystem; preserve
locks/*.guardsidecars while nodes run. Existing certificate, key, metadata and account paths remain compatible. - Custom backends: implement
move_keyfor no-clobber private-key archival. Implementcanonical_keywhen aliases exist. A backend accepting write fences must atomically implement guarded writes and moves; defaults reject unsupported fences before CA work. Pair manualtrack_lockcalls withuntrack_lock. - Remote certificate stores: pass the same coordinator instance to Config and RemoteCertStore. Etcd validates ownership during reference publication; FileStorage/Redis retain their documented coordination limits. Back up both references and blobs, and define retention for immutable history and archives.
- Crypto and builds: RSA key generation is opt-in (
rsa); default features use AWS-LC. Chooseringexplicitly when disabling defaults. Etcd builds needprotoc, including--all-featuresand etcd-backed remote-store examples. - HTTP and DNS: the HTTP/1.1 convenience wrapper accepts Content-Length bodies up to 1 MiB with a 30-second read deadline and rejects transfer encoding. Explicit DNS resolvers accept IPs (port 53) or socket addresses with custom ports; hostname resolver entries are rejected before lookup.
Quick start
use Arc;
use CancellationToken;
async
Prefer a high-level manager? CertManager wraps the same machinery behind a
cancellation-free facade:
let manager = builder
.issuers
.build?; // try_build() is the explicit fallible alias
manager.manage.await?; // foreground obtain + renew
let acceptor = new;
Cache::new starts renewal/OCSP maintenance automatically. Applications that
need an explicit lifecycle can use Cache::new_without_maintenance, then call
start_maintenance; stop_and_wait cancels and joins the task without
consuming the cache. A stopped cache cannot be restarted.
For a one-shot setup, certmagic::manage(&domains).await returns a ready
rustls::ServerConfig. For reusable policy and backend wiring, use
ConfigBuilder::new().policy(Policy::default()).storage(storage).build().
For a framework-neutral HTTP/1.1 wrapper with HTTP-01 handling and HTTPS
redirects, use certmagic::https or certmagic::https_on. These helpers advertise
HTTP/1.1 only, accept Content-Length bodies up to 1 MiB, reject transfer
encodings such as chunked, and apply a 30-second request-read deadline.
More in examples/: basic_https,
on_demand (handshake-time issuance),
custom_dns01 (wildcard via a custom DNS provider).
Integration with rustls
Three paths, pick per deployment:
| Path | API | Behavior |
|---|---|---|
| Sync cache resolver | Config::tls_config() |
Serves from the in-memory cache, no IO |
| Async acceptor | Config::certmagic_acceptor() |
Full async resolution (storage load / on-demand issuance / maintenance) before the handshake completes |
| Background remediation | automatic in tls_config() when on-demand is on |
Cache miss spawns issuance in the background; this handshake fails, the next one succeeds |
Feature flags
| Feature | Default | Enables |
|---|---|---|
file-storage |
✔ | FileStorage backend (atomic writes, heartbeat lockfiles) |
http-01 |
✔ | HTTP-01 challenge listener (Tokio TCP) |
dns-01 |
✔ | DNS propagation checks (hickory-resolver) |
ocsp |
✔ | OCSP stapling lifecycle (hand-rolled RFC 6960 codec) |
zerossl |
✔ | ZeroSSL ACME/EAB and REST API issuers |
local-cache |
Node-local read-through storage cache | |
redis-storage |
Redis values and owner-checked renewable leases | |
etcd-storage |
Etcd leases, snapshot reads and guarded publication | |
remote-cert-store |
Immutable certificate blobs with coordinated references | |
s3-cert-store |
S3 certificate blobs (caller-configured AWS SDK client) | |
vault-cert-store |
Vault KV v2 certificate blobs with CAS writes | |
secrets-manager-cert-store |
AWS Secrets Manager certificate blobs with pinned versions | |
rsa |
Opt-in RSA 2048/4096/8192 key generation | |
ring |
Ring crypto provider and x509-parser/verify |
|
aws-lc-rs |
✔ | AWS-LC crypto provider and x509-parser/verify-aws (including P-521 CSR signing) |
integration-tests |
Pebble end-to-end tests |
Choosing a storage backend
FileStorage remains the built-in durable backend. LocalCache is a node-local
read-through decorator, not a distributed source of truth. redis-storage adds
an optional Redis adapter. etcd-storage adds transaction-checked publication
for deployments using etcd for coordination. Optional S3, Vault KV v2 and AWS
Secrets Manager adapters store complete certificate resources through CertStore.
SQL storage is not bundled.
Custom backends implement Storage and Locker, then are supplied through
ConfigBuilder::storage. LockGuard::new accepts a backend-owned release
callback even without the file-storage feature. The callback must retain the
acquisition token and must not release another holder's lease. Network adapters
can override LockRelease::release_async for acknowledged cleanup. Use
guard.release_and_wait().await when acknowledgement matters; Drop remains a
best-effort fallback. guard.is_valid() reports advisory local lease health,
not write-side fencing. Automatic cleanup tracks acquisitions independently,
including identical names on separate backends.
Manual track_lock(storage, name) registrations must be paired with
untrack_lock(name) when ownership ends. A guard only unregisters its own
automatic registration; dropping an unrelated same-name guard no longer removes
manual ownership. Prefer the automatic acquisition helpers for new adapters.
LocalCache serves hits without awaiting backend operations and serializes
misses/writes per canonical key. Grouped reads, writes and moves acquire the
same gates in canonical sorted order, so independent groups overlap and
reversed/aliased key lists cannot deadlock. Prefix deletion excludes these operations;
concurrent misses for one key share the resulting fill. Cached entries retain
and reuse their key gates; eviction or the last in-flight operation reclaims
unused gates. Custom backends with aliases should override
Storage::canonical_key; its default preserves opaque keys, while FileStorage
and Redis normalize slash paths. Nested decorators forward this identity.
Single-key hits see outside changes only after eviction; grouped resource reads
delegate to the backend instead of mixing cached entries. Cancellation
cannot retract a dispatched backend write: after an ambiguous outcome, read
from the authoritative backend rather than assuming the local cache is current.
Exact key movement is a backend operation (Storage::move_key); it must not be
emulated using recursive delete. Canonical self moves are no-ops, descendants
survive, and an occupied destination returns StorageError::Conflict. A custom
Storage using KeyValueCertStore must implement this optional method to support
private-key archival; the default fails before any copy/delete. FileStorage
stages regular files in a private, fsynced sibling copy, publishes the complete
archive through a no-clobber hard link, then unlinks the source. Directory and
symlink sources are rejected. Archive copies remain owner-only on Unix even
when the imported source has broader permissions. The destination filesystem
must support hard links; unsupported filesystems fail with the source intact.
Interrupted filesystem moves may leave both names and are not a cross-file crash
transaction. Redis uses one atomic rename script; etcd uses revision comparisons.
Guarded etcd moves distinguish data conflicts from lost ownership inside one
transaction, keeping a valid lease available after an archive conflict.
Try-lock timeout budgets cover backend attempts as well as retry sleeps. Like other async deadlines, cancellation is cooperative and cannot retract a command already dispatched to a backend; expiring leases remain the recovery mechanism.
Certificate/private-key resources can use an independent CertStore through
ConfigBuilder::cert_store; accounts, challenge publications, locks and OCSP
remain on Storage. A transactional database or a versioned object bundle can
therefore provide stronger resource atomicity than the generic three-key adapter.
The generic adapter overlaps its three independent reads/existence checks; this
does not turn them into an atomic backend snapshot.
Redis is an optional choice for deployments already operating Redis, not a prerequisite for faster TLS cache hits. Configure persistence and eviction policy for durable account/private-key data; the adapter does not change server settings. Redis failover assumptions remain explicit. See Redis locking and persistence. For transactional storage, PostgreSQL locks or etcd transactions and leases are other building blocks. These capabilities still need a backend adapter; selecting a service alone does not provide fencing for this library's writes.
Reproducible local performance measurements and their limits are documented in benches/README.md.
Etcd adapter and guarded publication
Etcd support is available in 0.2.0. Enable
etcd-storage; builds enabling this feature (including --all-features) need
protoc on PATH. The optional client uses the crate's selected Ring/AWS-LC TLS
provider; default builds do not pull in the etcd/gRPC dependency graph.
use ;
let storage = connect.await?;
let config = builder.storage.build?;
All endpoints must belong to one cluster and use the same transport scheme.
HTTPS supports WebPKI roots or EtcdTlsOptions with custom CA/mutual-TLS PEMs.
Username/password credentials belong in options, not endpoint URLs. Debug and
backend error output omit credentials, PEMs and remote status bodies.
Config obtains/renews under its existing per-subject lock, then calls
CertStore::save_with_lock. The key-value adapter delegates all three certificate
components to Storage::store_tx_with_lock. Etcd compares the lock's revision,
lease ID and random token in the same transaction as those writes. Private-key
archival uses move_private_key_with_lock / move_with_lock, comparing source
revision and destination absence as well. An expired or replaced acquisition
cannot overwrite a newer certificate or archive/remove its private key.
LockRelease::write_fence provides an opaque backend context. Guards requiring a
fence are rejected by default implementations; a custom CertStore must support
both guarded operations explicitly. Config checks compatibility before CA work.
Etcd contexts work with clones/decorators of the originating storage handle;
a separately connected handle or a different backend is rejected. An etcd lock
combined with an arbitrary S3/secret-store CertStore does not provide a
cross-system transaction. The remote adapters below instead publish immutable
blob references through the originating coordinator.
Complete certificate reads use load_many and value-presence checks use
exists_exact_many; prefix-aware exists/exists_many retain their existing semantics.
Etcd performs each group in one transaction; LocalCache delegates bundle reads
together to avoid mixing cached generations. Generic backends retain their
parallel-read behavior. Prefix listing uses pagination at a fixed revision;
compaction errors are returned rather than silently switching snapshots.
Persistent values use a versioned binary envelope with a writer-clock modification timestamp; etcd revisions, not timestamps, determine ownership. Values never inherit the lock lease. Records/batch writes are limited to 1 MiB, with up to 64 guarded keys. Prefix deletion is an atomic exact-key plus child-range transaction. Prefix stat derives the newest observed child timestamp.
Leases default to 30 seconds with a 10-second keep-alive interval. Etcd TTL is
fixed at acquisition: explicit renewal refreshes it, but a request exceeding the
configured duration is rejected. Cancellation of an in-flight renewal discards
local ownership rather than reusing a delayed stream reply. Drop queues lease
revocation; release_and_wait awaits it, with TTL as the process/runtime-failure
fallback. Connection loss fails local lease health conservatively. Operations
may return timeout/unavailable errors during leader changes; ambiguous write
outcomes are not blindly retried. Linearizable writes require a quorum.
These guarantees cover guarded publication and private-key archival. Plain
store, delete, legacy save/move_private_key, account/challenge writes and
storage cleanup are not automatically fenced. FileStorage/Redis guards keep
their previous advisory-health behavior. This is not a guarantee against actors
that bypass guarded APIs or an atomic transaction spanning independent systems.
Run the explicit local integration lane (requires Docker and protoc):
Tests create uniquely named, owned containers/networks with loopback client ports,
and remove them on exit. They cover a three-member cluster, ownership replacement,
concurrent snapshot reads, Config obtain/renew publication, private-key archival,
keep-alive, cancellation/runtime loss, leader loss, quorum loss/recovery, password
authentication, and a separate mutual-TLS server. No CA or production endpoint is
contacted. The explicit example is examples/etcd_storage.rs.
Remote certificate stores
These adapters are available in 0.2.0. Enable s3-cert-store, vault-cert-store or
secrets-manager-cert-store; defaults remain unchanged. Each reuses
RemoteCertStore<B> and the ImmutableBlobStore contract for additional services.
Construct a blob backend, then call RemoteCertStore::new(backend, coordinator.clone(), namespace). Supply that same coordinator to
Config::builder().storage(coordinator).cert_store(certificates).
A complete certificate, private key and issuer metadata form one bounded, content-addressed blob. Only small references live in the coordinator; a cold load reads those references and fetches one blob. Guarded publication uploads first, then atomically checks the etcd owner and publishes references. A stale upload can leave an unused blob but cannot replace the current publication. FileStorage/Redis retain their existing weaker coordination guarantees. This protocol does not require a transaction spanning the two services.
Use one unique namespace per remote location and identical configuration on all
nodes. Back up both the coordinator and remote objects. Mixed references,
corrupt objects, missing referenced blobs and permission failures return errors.
Legacy administrative save/remove calls remain unfenced. Private-key archival
moves a reference without overwriting an existing archive;
load_archived_private_key(destination) resolves it. Removal and archival are
logical operations: immutable objects, older secrets and failed-upload remnants
are retained. There is no automatic garbage collector or physical key erasure;
retention must account for live references, archives and uncertain write outcomes.
| Adapter | Configuration and operational contract |
|---|---|
S3CertStore / S3BlobStore |
Existing bucket supporting If-None-Match: *; default encoded limit 1 MiB. Configure encryption and access policy on the bucket. Uses bounded GET for existence to distinguish a missing object from a missing bucket. |
VaultCertStore / VaultKv2BlobStore |
KV v2, CAS=0, immutable version 1, HTTPS by default; default decoded blob limit 256 KiB. Token requires create/update/read on data paths and read on metadata paths. Disable scheduled version deletion; token renewal belongs to the caller. |
SecretsManagerCertStore / SecretsManagerBlobStore |
One secret per blob, pinned version ID, encoded limit 64 KiB including base64 PEM and metadata. Plan secret-count quotas and ongoing storage cost; IAM should deny changes/deletion of retained versions. Optional KMS encryption key; private keys remain exportable, not HSM signing keys. |
AWS adapters accept caller-configured SDK clients. Set credentials, region,
retry policy and HTTP transport explicitly; cert_store::remote::aws_http_client()
uses the selected Ring/AWS-LC provider. The adapters do not discover ambient
credentials. S3 and Vault bound downloaded bodies; Secrets Manager checks the
service's 64 KiB binary limit after the SDK has decoded its JSON response, so
custom endpoints must be trusted to enforce service-compatible response limits.
Constructors validate local options, not remote availability or IAM policy.
Runnable configuration examples use etcd for coordination and issue no
certificates: s3_cert_store,
vault_cert_store, and
secrets_manager_cert_store.
Compile with the corresponding adapter feature plus etcd-storage (requires
protoc). Protocol tests use loopback mocks; explicit Docker tests use isolated
MinIO, Vault dev mode and LocalStack, never production credentials. MinIO's
community distribution is source-only;
the test Dockerfile builds the official release's fixed commit instead of
pulling the unavailable upstream image:
Redis adapter
Redis support is available in 0.2.0. Enable redis-storage explicitly:
= { = "0.2", = ["redis-storage"] }
use ;
# async
The adapter supports a single Redis endpoint, password/ACL URL authentication,
rediss:// with certificate verification and redis-rs Unix socket URLs. It uses
redis-rs connection multiplexing/reconnection, not one connection per operation.
Defaults are a 30 s lease, 10 s heartbeat, 5 s command timeout and 100 ms lock poll.
The heartbeat interval plus command budget must be shorter than the lease.
Values and modification timestamps are atomic within one Redis hash and have
no TTL. Lock keys occupy a separate namespace and use SET NX PX; Lua scripts
verify the acquisition token before renewal or deletion. Explicit extension is
not shortened by a later heartbeat. Prefix operations use escaped SCAN patterns
and bounded deletion batches; concurrent prefix changes are not atomic snapshots.
This is a single-endpoint lease adapter, not Redlock or write fencing. Redis
Cluster and Sentinel discovery are not implemented. Losing a heartbeat fails
local lease health closed. Issuance/renewal checkpoints stop retries and
publication after detected lease loss; callers needing fenced writes still
require a stronger atomic write protocol. CertStore remains separate, and the generic three-key resource
adapter is not a crash-atomic transaction. Persist keys/accounts appropriately
(e.g. AOF with an explicit fsync policy and a non-evicting capacity plan), use
ACL/TLS as appropriate, and do not log connection URLs. No automatic server
configuration changes are performed.
Example: cargo run --example redis_storage --features redis-storage.
Tests start owned temporary Redis instances on loopback ports; they never use a
configured production Redis URL:
Set CERTMAGIC_REDIS_SERVER to an alternate local server binary. Network tests
are ignored by default; the pure configuration test runs normally. Local evidence
uses Redis 8.10.2, including AOF restart, authentication, lease loss, cancellation
and old-holder protection. TLS handshake, Cluster/Sentinel, Valkey and Dragonfly
are not claimed as validated by those tests.
Other backend candidates
| Backend | Suitable integration | Required work / current status |
|---|---|---|
| Valkey | Reuse the Redis protocol adapter | Candidate; run the same compatibility tests before claiming support |
| etcd | Storage + Locker, guarded transactions and snapshot reads |
Implemented behind etcd-storage; protected writes require the originating backend context |
| Consul KV | Storage + session-based Locker |
Separate adapter; account for session invalidation and lock-delay |
| DynamoDB | Conditional writes for storage and lease records | Separate adapter; TTL deletion is asynchronous, so expiry must be checked in conditions |
| redb / RocksDB | Embedded single-node storage | Separate adapter; move blocking work off Tokio and do not imply distributed locking |
| S3 / Vault KV v2 / AWS Secrets Manager | Complete-resource CertStore |
Implemented; retain shared coordination for references, accounts and challenges |
See Valkey compatibility, etcd APIs, Consul sessions, and DynamoDB expiry semantics. FileStorage, RedisStorage, EtcdStorage and the three remote CertStore adapters are implemented here. Valkey, Consul, DynamoDB and embedded KV remain candidates.
File storage coordination
FileStorage serializes lock creation, heartbeats, release, and stale takeover
using permanent locks/*.guard sidecars and unique holder IDs. The shared
filesystem must support OS file locks; never delete the sidecars while an
instance is running. Stop all instances before upgrading from the earlier
heartbeat-only protocol, then restart them on the same version.
This is a lease protocol, not a fencing service: a process paused past lease
expiry can resume application writes after another instance takes over.
Deployments requiring fencing must supply a backend that enforces it.
Certificate resources still use separate certificate/key/metadata writes;
store_tx rolls back reported errors but is not a crash-atomic transaction.
Testing
# Real ACME end-to-end against Pebble (Let's Encrypt's reference server):
PEBBLE_CHALLENGE=http-01
PEBBLE_CHALLENGE=tls-alpn-01
# Deterministic boundary check (forces Cargo offline mode):
# Explicit local Pebble lane (never a production CA test):
# Listener/handler checks on ephemeral loopback ports (no CA or public DNS):
The Pebble test exercises the full lifecycle — account registration, selected
HTTP-01/TLS-ALPN-01/DNS-01 validation, issuance, storage persistence, and
forced renewal — and is the reason several protocol details (lowercase
application/jose+json, optional challenge tokens, order ready → finalize
ordering) are correct. HTTP-01 and TLS-ALPN-01 modes disable challtestsrv's
canned challenge responders so the certmagic solver itself serves Pebble's
validation request on the configured high port.
Project layout
src/
├── acme/ # ACME protocol layer: transport, JWS, directory, orders, issuer
├── solvers/ # http-01 / tls-alpn-01 / dns-01 / distributed challenge solvers
├── storage/ # Storage/Locker traits + FileStorage (atomic writes, locks)
├── ocsp/ # RFC 6960 codec + stapling lifecycle
├── certificate.rs # parsing, name matching, subject qualification, renewal math
├── cache.rs # in-memory certificate cache
├── config.rs # orchestration: obtain / renew / manage / revoke
├── handshake.rs # handshake-time issuance + on-demand gating
├── tls_integration # rustls glue (three integration paths)
└── runtime.rs # retry budget, job manager, single-flight
License
Apache-2.0 — see LICENSE.
Acknowledgments
- Special thanks to caddyserver/certmagic — this project follows its design.
- Thanks to salvo-rs/certon — its API naming inspired the compatibility aliases and high-level manager shipped alongside the native API.