ferryman-edge

Programmable mTLS L7 reverse proxy in Rust — extends ferryman (P2) with rustls 0.23 + aws-lc-rs mTLS termination, in-line RS256 JWT validation with an LRU verification cache, per-tenant GCRA rate limiting, and a SIGUSR1-driven hot-reload that flips cert chains and routing tables without dropping live connections. Pingora-class chops at miniature scale.

The problem

Real edge proxies do mTLS termination, JWT validation in-line, per-route rate limiting, and cert hot-reload — and they make defensible decisions on body buffering, HTTP/2 stream control, and connection pooling. ferryman-edge is the project that gets the Cloudflare Pingora team to reply.

Architecture (delta on top of P2)

                Client (with mTLS cert)
                       |
                       v
              +--------+---------+
              | rustls TLS+mTLS  |  cert reload via SIGUSR1
              | + JWT validate   |  (no connection drops)
              +--------+---------+
                       |
                       v
              +--------+---------+
              | per-tenant rate  |  governor crate, keyed by JWT.sub
              | limit (per route)|
              +--------+---------+
                       |
                       v
              +--------+---------+
              | RouteService     |
              | (P2's table)     |
              +--------+---------+
                       |
                       v
              +--------+---------+
              | hyper client     |  pool: 1 conn per upstream * N
              | HTTP/2 multiplex |  feature flag: boxed_body vs collected
              +--------+---------+
                       |
                       v
                  upstream svc

Stack

Pinned versions live in Cargo.toml.

Install

cargo install ferryman-edge      # installs the `ferryman-edge-server` binary

The reusable pieces (TLS reload, JWT verifier, rate limiter, routing + circuit breaker) are published separately as ferryman-edge-core.

Examples

• examples/edge-demo runs the proxy in front of sample services and exercises every feature end to end: mTLS, JWT claims, tenant propagation, routing, body limits, rate limiting, circuit breaking, hot reload of routes and certificates, metrics, and graceful shutdown. One command: examples/edge-demo/run.sh.
• examples/embed-core adds mTLS, JWT auth and per-tenant rate limiting to your own axum service with ferryman-edge-core, no proxy hop.

Quick start

# 1. Generate a test mTLS chain (root + 3 intermediates + server + 1 client + RSA JWT keypair).
./scripts/gen-test-certs.sh

# 2. Build + run against the example config (mTLS terminate on :8443).
cargo run -p ferryman-edge -- --config config.toml

# 3. Call with a valid client cert + an RS256 JWT signed by certs/jwt-priv.pem:
curl -i --cacert certs/ca.crt \
        --cert   certs/client.crt \
        --key    certs/client.key \
        -H "Authorization: Bearer $(scripts/mint-jwt.sh tenant-a 3600)" \
        https://localhost:8443/svc-a/hello

# 4. Hot-reload cert + routing table atomically:
kill -USR1 $(pidof ferryman-edge-server)

Configuration reference, reload, metrics and troubleshooting: docs/operations.md.

Request pipeline

Every request passes the same gates, in order:

On the way through, the proxy strips hop-by-hop headers (both directions, including any named in Connection), then stamps x-ferryman-tenant: <sub> (any client-supplied value is dropped first). It rewrites Host to the upstream, replaces x-forwarded-for with the peer IP (dropping client-sent Forwarded / X-Real-IP), sets x-forwarded-proto: https, and downgrades the outbound request to HTTP/1.1. Inbound protocol is pinned from ALPN (h2 or http/1.1). A connection with no request within 10 s of the handshake is closed (this also covers a stalled h2 preface); h2 connections then get keep-alive pings and a 64-stream cap.

Each upstream has a Closed / Open / HalfOpen circuit breaker (ferryman_circuit_state{upstream}: 0/1/2; cooldown_secs must be ≥ 1). A transport error, a 502–504, or a timeout opens it — under boxed_body a timeout only counts if the client had finished uploading; after cooldown_secs exactly one request is let through as the probe. A plain 500 does not trip it, and neither does a failure caused by the client's own body (size cap, disconnect). The active health checker (GET <upstream>/health every health_interval_secs) opens and closes it too. A route reload keeps breaker state for rules whose prefix, upstream, and cooldown are unchanged.

SIGTERM / SIGINT stop accepting and drain in-flight connections for up to 25 s. SIGUSR1 reloads TLS material and the routing table; the JWT settings and tenant_rps are read once at boot.

Set [jwt] issuer and audience for anything beyond local dev — without them, any token signed by the issuer key is accepted, whichever service it was minted for.

Design tradeoffs

P4 makes three decisions worth defending in a hiring loop.

(a) boxed_body is off by default

Cargo feature boxed_body swaps the upstream client to http_body_util::BoxBody and streams request and response bodies. With it off, the proxy collects each body once into a Full<Bytes> before forwarding. Both builds enforce the 8 MiB request cap; under streaming, a chunked upload with no Content-Length that exceeds it is cut mid-stream and answered 413, without counting against the upstream's breaker. Streaming mode has no separate body-read deadline: a slow upload runs inside the upstream's 30 s budget and ends as 504.

Estimates from the design spec (not yet measured here): boxed adds ~200 µs per request at 10 MB; collected adds ~80 µs at 1 KB but allocates ~req_size. For an internal proxy fronting JSON APIs under ~256 KB the collected path wins on code complexity, allocator pressure (because the JSON allocator already paid the cost), and steady-state latency. Pingora picks streaming for general-purpose CDN traffic where payloads skew big and bimodal; that calculus inverts for an internal API edge. Flip the feature on (cargo build --features boxed_body) when your p99 latency tells you the collect-first cost dominates.

(b) SIGUSR1 reload over filesystem-watch

P2 (ferryman) reloads its routing table via notify filesystem events. P4 deliberately swaps to SIGUSR1 because:

• k8s mounts ConfigMaps via a symlink-swap dance. notify reports this as a chain of remove + create events on the symlink target, not the watched path. Without per-platform special-casing the watcher silently misses the reload — a worst-case failure mode for a security-sensitive hot-swap of TLS material.
• Editors emit a parade of Modify events for in-place writes that have no business triggering a reload (cursor moves, autosave drafts).
• SIGUSR1 is one POSIX call with predictable semantics across every deploy target. The operator runs kill -USR1 $(pidof ferryman-edge-server) after kubectl rollout restart of the ConfigMap, or wires it into cert-manager's renewal hook.

Both cert reload (tls::ReloadingTls) and route reload (server/src/reload.rs) share the same signal — one trigger swaps both surfaces atomically from the operator's perspective.

(c) rustls + aws-lc-rs over OpenSSL

• Pure-Rust audit story. rustls is the only TLS stack with a clean memory-safety argument all the way to the cipher implementations (aws-lc-rs is the AWS-libcrypto Rust binding; ring is the historical alternative). For an edge proxy that terminates customer-data TLS, that argument matters more than the C/Go ecosystem's parity.
• FIPS path. aws-lc-rs has a FIPS-mode build via the same crate. No swap-out at deploy time, no separate provider — flip a feature flag and recompile. OpenSSL FIPS 3.0 modules ship, but the build process is brittle and OS-distribution-specific.
• Cost. Distroless final image instead of scratch (aws-lc-rs needs libc + dynamic loader). ~12 MB extra over a musl/scratch build. Worth it for the audit + FIPS leverage.

Benchmarks

# JWT verify: cache hit vs miss (criterion).
cargo bench -p ferryman-edge-core --bench jwt_verify

# Zero-loss reload check: curl workers (fresh mTLS handshake per request)
# while SIGUSR1 fires at 50% and 75% of the run. Needs the server up and
# an upstream answering /svc-a/echo.
./benches/reload.sh 60 8

wrk/wrk2 cannot present a TLS client certificate, so benches/wrk2.lua only works against a listener without mTLS; the 50k rps target below needs an mTLS-capable load generator and is not measured yet.

Repository layout

ferryman-edge/
  Cargo.toml                       # workspace
  config.toml                      # example: TLS paths, JWKS, 2 upstreams, rps cap
  crates/
    core/                          # tls, jwt, ratelimit, route, health, config (+ jwt_verify bench)
    server/                        # accept loop + auth middleware (lib), proxy, reload, e2e tests
  certs/                           # generated, gitignored (see scripts/gen-test-certs.sh)
  scripts/
    gen-test-certs.sh              # root + 3 intermediates + server + client + JWT keypair
    mint-jwt.sh                    # RS256 token signed by certs/jwt-priv.pem (openssl only)
  benches/
    wrk2.lua                       # throughput script (non-mTLS listeners only)
    reload.sh                      # zero-loss reload check (curl workers + kill -USR1)
  Dockerfile                       # cargo-chef multi-stage, distroless final
  fly.toml                         # Fly.io 2-region (sin + iad)
  deny.toml                        # cargo-deny config
  rust-toolchain.toml              # stable channel
  .github/workflows/ci.yml         # fmt + clippy + nextest + deny + criterion + mTLS smoke
  docs/
    operations.md                  # config reference, reload, metrics, troubleshooting
    publishing.md                  # crates.io release plan + checklist
  examples/
    edge-demo/                     # run the proxy: PKI, tokens, config, backends, 11 scenarios
    embed-core/                    # ferryman-edge-core inside your own axum service
  book/                            # mdBook site (GitHub Pages); includes README, docs/, CHANGELOG
    specs/2026-05-28-ferryman-edge-design.md
    plans/2026-05-28-ferryman-edge-phase-1-scaffold.md
  PROGRESS.md
  CHANGELOG.md

Roadmap

Phase 1 (scaffold) and Phase 2 (mTLS + JWT + rate limit on the real request path, circuit breaker, streaming boxed_body, e2e tests) are done. Open: mTLS-capable throughput numbers against the 50k rps / p99 targets, and the Fly.io 2-region deploy. See PROGRESS.md.

License

Dual-licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT License (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.