⚡ arcly-http

NestJS ergonomics meets Rust safety & speed

A batteries-included, enterprise-grade HTTP framework built on axum — declarative controllers, zero-lock DI, a complete auth pipeline, realtime gateways, and first-class observability.

Quick Start • Features • Architecture • Auth Pipeline • Examples • Configuration

---

👀 At a Glance

use arcly_http::prelude::*;

pub struct UserController;

#[Controller("/users", tags("users"))]
impl UserController {
    /// Fetch a single user by ID — JWT-protected, traced, documented.
    #[Get("/:id", summary("Get user"), security("bearer"))]
    #[UseInterceptors(TraceInterceptor)]
    async fn get_user(
        svc: Inject<UserService>,          // ← zero-lock DI, O(1) resolution
        ctx: RequestContext,               // ← claims & session pre-decoded
        #[Param("id")] id: u64,            // ← typed path extraction
    ) -> Result<Json<User>, HttpException> {
        JWT_AUTH.check(&ctx)?;             // ← zero-overhead guard
        svc.find(id).map(Json).ok_or_else(|| NotFound::new("user not found").into())
    }
}

#[Module(controllers(UserController), providers(UserService))]
pub struct AppModule;

#[tokio::main]
async fn main() -> std::io::Result<()> {
    App::launch::<AppModule>("0.0.0.0:3000").await   // /docs, /healthz, /metrics — free
}

One annotation gives you: routing, OpenAPI docs, JWT decoding, distributed tracing, and dependency injection — with no locks on the request hot path.

---

📦 Feature Matrix

	Capability	Details
🧩	Declarative routing	#[Controller], #[Get]/#[Post]/#[Put]/#[Patch]/#[Delete], typed #[Param]/#[Query]/#[Body] extraction
🎛️	Production governance	One LaunchConfig (+ ARCLY_* env overrides): request deadline (504), in-flight admission cap (503 + Retry-After), panic boundary (500, never a dropped connection), 413 body cap, bounded response cache + sweeper, WS drain deadline, drain-aware /readyz, config-driven CORS, x-request-id everywhere — all lock-free
🔌	Zero-lock DI	#[Injectable] + #[Module] DAG → frozen &'static container; Inject<T> resolves in O(1) with no locks, no allocation
🔐	JWT auth	JwtService (HS/RS/ES families), access + refresh pairs, automatic Bearer decoding at the boundary, JWT_AUTH / RoleGuard
🍪	Signed cookies	CookieService — HMAC-SHA256, tamper-proof; boundary falls back to the JWT cookie when no Bearer header (browser-friendly, no localStorage)
🗂️	Server-side sessions	SessionManager + pluggable SessionStore trait; ctx.session() on every request; SESSION_AUTH guard
🌐	OAuth 2.0 + PKCE	OAuth2Provider trait + registry — Authorization Code flow with atomic single-use CSRF state; Google & GitHub providers in the example
🛡️	Fine-grained permissions	resource:action perms embedded in JWT (issue_access_with_perms) → zero-I/O permission checks with wildcard support (users:*)
🏢	Multi-tenancy	TenantRegistry (header / subdomain strategies, frozen map) resolved once per request; ctx.tenant() + TENANT guard cross-checks the JWT tenant claim against the resolved tenant
🌍	Distributed rate limiting	DistributedRateLimit over a pluggable RateLimitBackend — cluster-wide sliding window (atomic Redis Lua), sub-then-IP principal keying, explicit FailOpen/FailClosed policy
🔄	Secrets rotation	SecretSource (Vault / AWS SM / env) + Rotating<T> over ArcSwap — hot-swap JWT_SECRET/cookie secrets with zero restarts; previous key verifies through the grace window
🗄️	Datasource routing	DataSource trait + frozen DataSourceRegistry: tenant-scoped pools, read/write splitting, ReadAfterWritePin defeats replica lag
📋	Audit trail	#[AuditLog(action, resource)] — one record per mutation (who/what/when/where/outcome), lock-free try_send hot path, hash-chained (SHA-256) batches into an app-provided append-only AuditSink
📤	Transactional outbox	OutboxTx::enqueue inside the business transaction + background OutboxRelay (poll → publish → ack, at-least-once with idempotency keys) — kills the dual-write problem
🏷️	API versioning	#[Version("v1")] mounts the controller under /v1/...; #[Deprecated(sunset = "…")] adds RFC 8594 Deprecation/Sunset headers so clients learn deadlines on the wire
🧬	Unified DB layer	ArclyDbPool — one injectable handle over SQLx / SeaORM / Diesel (Cargo features db-sqlx-* / db-seaorm-* / db-diesel-*); tenant routing + RW splitting apply via the existing registry; sync Diesel fully isolated behind spawn_blocking
💾	#[Transactional]	Begin on the tenant pool → commit on Ok, rollback on Err / #[Timeout] cancellation; task-local tx (no locks); with_current_tx reaches it from services — outbox rows ride the same transaction
🔏	Distributed lock	DistributedLock + DLockBackend — fencing tokens (cluster-monotonic), compare-and-delete release, auto-renew heartbeat; OutboxRelay leader election ships on it
🗃️	DB migrations	MigrationRunner — versioned arcly_migrations ledger per datasource, checksum drift = boot failure, fleet DDL serialized on the cluster lock, multi-tenant run_all
🔁	Idempotency keys	#[Idempotent(ttl = "24h")] — Stripe-style Idempotency-Key: atomic claim, stored-response replay (Idempotency-Replayed: true), 409 for concurrent duplicates
⚖️	ABAC policy engine	#[RequirePolicies("orders.refund")] — default-deny attribute rules (subject × resource × env) over ArcSwap; hot-reload a new PolicySet and the next request obeys it — no OPA call on the hot path, no restart
📨	Event consumer mesh	#[EventConsumer] + #[EventPattern("topic")] — link-time handler registry (frozen dispatch), consume-side dedupe, typed failure fates (EventError::Retry vs DeadLetter — poison messages park immediately), configurable batch concurrency, drain-aware; transport (Kafka/AMQP/NATS) is one app-side trait
🧪	Testing harness	testing::TestRequest builds a production-shaped RequestContext through the real boundary pipeline (claims, tenant, trace, DI) for unit-testing guards/interceptors/handlers; testing::TestServer boots the app on an ephemeral port for integration tests
🧵	Cross-channel tracing	traceparent rides outbox rows and message envelopes — consumers continue the producing request's trace (TraceContext::from_traceparent) instead of starting orphan roots
🕶️	PII masking	#[MaskFields("card:last4")] + hot-reloadable Masker (ArcSwap) — Redact/Hash/Last4/Drop applied before durability at every sink: responses, the idempotency replay cache, outbox rows, and dead-lettered payloads; fields surface as x-arcly-masked-fields
🔏	Field-level encryption	#[EncryptFields(key = "tenant:acme", fields("ssn"))] — AES-256-GCM envelope encryption under per-tenant / per-subject DEKs (CryptoVault, ArcSwap key ring, KMS behind the KekSource trait); rotation without re-encryption, and GDPR erasure via crypto-shredding (vault.shred) even inside append-only sinks
🏬	Dynamic tenants	TenantRegistry over ArcSwap snapshots — upsert/suspend/resume live on the next request (no redeploy); suspended tenants get 401 with no silent fallback
⏱️	Deadlines & bulkheads	#[Timeout("2s")] → 504 + future cancellation (worker freed); Bulkhead (const semaphore) sheds with 503 instead of queueing when a slow dependency saturates
✅	Validation	validator integration on #[Body]/#[Query] — violations return RFC-7807 422 with per-field errors, automatically
⚡	Resilience	#[circuit_breaker] — lock-free state machine with single-probe half-open + fresh-cooldown re-open, transitions observable per method; RateLimit fixed-window limiter (single CAS); Bulkhead admission control — all with Prometheus counters/gauges
📡	Realtime	WebSocket gateways with rooms & broadcast, JWT auth at handshake; SSE streams
🔭	Observability	W3C traceparent in/out, Prometheus /metrics, /healthz + /readyz with pluggable checks, OTLP traces, structured JSON logs
📜	OpenAPI	Swagger UI at /docs — and the spec mirrors the whole hardening stack: Idempotency-Key param + 409 + replay header, ABAC-aware 403s, 504 with the route deadline, RFC 8594 Sunset headers, x-arcly-* vendor extensions for gateways/SDK generators
🧱	Plugin system	ArclyPlugin lifecycle (init → start → draining → graceful drain, per-plugin budget via LaunchConfig); plugin routes share the exact same request pipeline as macro routes; global interceptors, pre-body BoundaryFilters, explicit DI overrides, runtime-mounted routes under /_plugins/* (ArcSwap, lock-free)
🚀	Response caching	#[CacheTTL(30)] + CacheInterceptor — keyed on path + query, hit/miss stats, one-call flush

---

🏛 Architecture

flowchart LR
    subgraph entry ["Request Entry Points"]
        A["HTTP<br/>macro routes"]
        B["Plugin<br/>routes"]
        C["WebSocket<br/>handshake"]
    end

    subgraph pipeline ["One Shared Pipeline"]
        D["auth::extract<br/>Bearer → cookie → session"]
        E["observability::propagation<br/>W3C traceparent"]
        F["RequestContext"]
    end

    subgraph runtime ["Runtime"]
        G["Guards<br/>JWT · Role · Session · Perms"]
        H["Handler"]
        I["core: frozen DI<br/>(HTTP-agnostic)"]
    end

    A --> D
    B --> D
    C --> D
    D --> F
    E --> F
    F --> G --> H
    H -. "Inject&lt;T&gt; · O(1), no locks" .-> I

Workspace layout

arcly-http/                  ── the framework crate
└── src/
    ├── auth/                identity & access — one place, one pipeline
    │   ├── jwt.rs           JwtService: sign / decode / validate (rotating keys)
    │   ├── cookie.rs        HMAC-SHA256 signed cookies (rotating secrets)
    │   ├── session.rs       SessionManager + SessionStore trait
    │   ├── oauth.rs         OAuth2Provider trait + registry
    │   ├── secrets.rs       SecretSource · Rotating<T> (ArcSwap) · watcher
    │   ├── policy.rs        ABAC engine: default-deny PolicySet · hot reload
    │   ├── guards.rs        JWT_AUTH · RoleGuard · SESSION_AUTH
    │   └── extract.rs       ★ the single credential-extraction pipeline
    ├── core/                DI engine + plugin lifecycle (HTTP-agnostic)
    ├── pipeline/            ★ unified Provenance: trace + tenant + credentials,
    │                          ONE extraction shared by HTTP · WS · event mesh
    ├── data/                DataSource · tenant registry · RW splitting · outbox
    │   ├── db.rs / tx.rs    unified ArclyDbPool + #[Transactional] runtime
    │   ├── drivers/         sqlx · seaorm · diesel (feature-gated)
    │   └── migrate.rs       versioned migrations · checksum drift guard
    ├── web/                 boundary (run_entry), RequestContext, governor, CORS,
    │                        security headers, validation, tenants, idempotency
    ├── compliance/          data governance: #[MaskFields] masking · #[EncryptFields] envelope encryption + crypto-shredding
    ├── messaging/           event consumer mesh: #[EventPattern] · retries · DLQ · event bus
    ├── realtime/            WebSocket gateways + SSE
    ├── observability/       telemetry, metrics, health, propagation, audit trail
    ├── resilience/          circuit breaker · rate limiting · timeout · bulkhead · dlock
    └── docs/                OpenAPI spec assembly + Swagger UI

arcly-http-macros/           ── proc-macros (#[Controller], #[Module], …)
examples/
├── enterprise_app/          ── full e-commerce backend, every feature exercised
│   └── src/
│       ├── api/             one file per resource + shared guards
│       ├── auth/            AuthService, OAuth providers, permissions
│       ├── infra/           bootstrap, redis, sessions, datasources, secrets
│       └── domain.rs · services.rs
└── realtime_enterprise_app/ ── realtime-focused walkthrough

Design invariants

These are enforced by structure, not convention — each has exactly one home in the tree.

Invariant	Why it matters
One request pipeline — every transport (HTTP, WS handshakes, event mesh) extracts trace + tenant + credentials through pipeline::Provenance	A security fix lands everywhere at once; transports can never drift (WS gets the same tenant enforcement as HTTP, suspended tenants' queued events stop processing)
core is HTTP-agnostic — the DI engine never imports axum	Engine internals evolve without touching routing code
Handlers never see axum — RequestContext is the only request surface	Proven in 0.2.0: the axum 0.7→0.8 upgrade changed zero user-facing route syntax (:id stays; arcly translates to {id} at mount)
Macros target one stable module — codegen references __macro_support only	Internal files move freely; the macro crate never needs to follow
No locks on the request path — frozen maps, atomics, and ArcSwap only	Tenancy, rate limiting, and secret rotation scale without contention

---

🚀 Quick Start

git clone git@gitlab.com:arcly/arcly-http/arcly-http.git
cd arcly-http

# Run the full-featured example — works without Redis (in-memory fallback)
cargo run -p enterprise_app

Endpoint	URL
📜 Swagger UI	http://localhost:3000/docs
❤️ Health	http://localhost:3000/healthz
📊 Metrics	http://localhost:3000/metrics
📡 Live SSE feed	curl -N http://localhost:3000/streams/inventory

---

🔐 Authentication & Authorization

The enterprise example seeds two demo users at startup:

Email	Password	Role	Permissions
admin@example.com	admin123	admin	users:* products:* orders:* admin:*
user@example.com	user123	customer	products:read orders:read orders:create

Three ways in, one pipeline

# ── 1. Login: returns a JWT pair AND sets signed cookies ──────────────────────
curl -s -X POST http://localhost:3000/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"admin@example.com","password":"admin123"}' -c cookies.txt | jq

# ── 2a. Bearer auth (API clients) ──────────────────────────────────────────────
TOKEN=$(curl -s -X POST http://localhost:3000/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"admin@example.com","password":"admin123"}' | jq -r .access_token)
curl -s http://localhost:3000/auth/me -H "Authorization: Bearer $TOKEN" | jq

# ── 2b. Cookie auth (browsers) — no header, the signed cookie carries the JWT ──
curl -s http://localhost:3000/auth/me -b cookies.txt | jq

# ── 3. Tampered cookies are rejected by HMAC verification ─────────────────────
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3000/auth/me \
  -H "Cookie: arcly_auth=forged.signature"        # → 401

Token lifecycle guarantees

sequenceDiagram
    participant C as Client
    participant S as Server
    participant R as Redis

    C->>S: POST /auth/login
    S->>R: SET refresh::{jti} (TTL 7d)
    S->>R: SET access2refresh::{access_jti} → refresh_jti
    S-->>C: token pair + Set-Cookie ×2

    C->>S: POST /auth/refresh
    S->>R: GETDEL refresh::{jti}  (atomic — single use)
    Note over S,R: concurrent replay of the same<br/>refresh token loses the race → 401
    S-->>C: new token pair

    C->>S: POST /auth/logout
    S->>R: GETDEL access2refresh::{access_jti}
    S->>R: DEL refresh::{refresh_jti}
    S-->>C: 204 + Max-Age=0 cookies (auth + session cleared)

• Refresh tokens are single-use — rotation uses atomic GETDEL, so a stolen-then-replayed token always loses the race.
• Logout actually revokes — the access→refresh JTI mapping lets a Bearer-only logout kill the paired refresh token, delete the server-side session, and expire both cookies.
• Permissions ride the token — PermissionGuard::require("users:read") checks the JWT perms claim first (zero I/O), with a role-map fallback. Wildcards (users:*) are honored.
• Login is brute-force-protected cluster-wide — /auth/login sits behind a DistributedRateLimit (10/min per principal, fail-closed): the sliding window counts in Redis via one atomic Lua call, so the limit holds across every replica.
• Secrets rotate without restarts — a SecretSource watcher hot-swaps JWT/cookie keys through ArcSwap; the previous key keeps verifying until live credentials expire naturally.

Declarative route hardening

Every cross-cutting policy is one attribute on the handler — the macro weaves it into the same compiled thunk, so there is no runtime middleware lookup:

#[Post("/", status(201), security("bearer"))]
#[Idempotent(ttl = "24h")]                                // safe client retries
#[MaskFields("payment.card:last4")]                       // PII never persists raw
#[RequirePolicies("orders.create")]                       // ABAC gate (hot-reloadable)
#[AuditLog(action = "order.create", resource = "order")]  // compliance record
#[Timeout("5s")]                                          // 504 + cancel on expiry
#[UseInterceptors(TraceInterceptor)]
async fn create_order(ctx: RequestContext, #[Body] dto: CreateOrderDto) -> … {
    JWT_AUTH.check(&ctx)?;
    let _permit = ORDER_BULKHEAD.try_enter()?;            // 503 when saturated
    …                                                     // outbox enqueue is
}                                                         // atomic with the write

# Versioned + deprecated endpoints announce their lifecycle on the wire:
curl -si http://localhost:3000/v1/meta/info | grep -iE "deprecation|sunset"
# deprecation: true
# sunset: 2027-01-01

And the stack is self-documenting — every attribute above surfaces in the generated OpenAPI spec, no extra annotations:

// curl -s localhost:3000/openapi.json | jq '.paths["/orders/"].post'
{
  "parameters": [ { "name": "Idempotency-Key", "in": "header", … } ],
  "responses": {
    "409": { "description": "Conflict — a request with this Idempotency-Key is already in flight" },
    "504": { "description": "Gateway Timeout — handler exceeded its 5000ms deadline (work cancelled)" },
    "201": { "headers": { "Idempotency-Replayed": { … } } }
  },
  "x-arcly-idempotent-ttl-secs": 86400,
  "x-arcly-audit": { "action": "order.create", "resource": "order" },
  "x-arcly-timeout-ms": 5000
}
// ABAC routes get policy-aware 403s + x-arcly-policies; sunset routes get
// deprecated:true, x-sunset, and documented RFC 8594 response headers.

Multi-tenancy & datasource routing

# Tenant resolved once per request (X-Tenant-Id header → frozen TenantRegistry);
# TENANT guard rejects a forged header that contradicts the JWT tenant claim.
curl -s http://localhost:3000/admin/tenant-info \
  -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: acme" | jq

{
  "tenant": "acme",
  "datasource": "acme",
  "read_before_write": "acme-replica-0",   // reads → replica (round-robin)
  "write":             "acme-primary",     // writes → primary (trips the pin)
  "read_after_write":  "acme-primary"      // pinned — replica-lag protection
}

OAuth 2.0 (Google / GitHub)

Providers self-register only when credentials are present — no config, no route:

export GOOGLE_CLIENT_ID=…  GOOGLE_CLIENT_SECRET=…
export GITHUB_CLIENT_ID=…  GITHUB_CLIENT_SECRET=…
export OAUTH_REDIRECT_BASE=https://your-domain.example

GET /oauth/{provider}/authorize   → { "url": "https://accounts.google.com/…" }
GET /oauth/{provider}/callback    → validates CSRF state (atomic, single-use),
                                    exchanges code with PKCE, upserts the user,
                                    mints tokens through the same path as
                                    password login, sets both cookies

---

Tenants are dynamically provisionable: POST /admin/tenants upserts a tenant live (effective on the next request — no redeploy), and /admin/tenants/:id/suspend hard-cuts its traffic to 401 with no silent fallback to the shared pool. The hot path stays one ArcSwap pointer load.

Unified database layer & #[Transactional]

Pick your ecosystem with Cargo features — the injected handle and the transaction semantics stay identical:

arcly-http = { version = "0.1", features = ["db-sqlx-postgres"] }
# or: db-seaorm-postgres · db-diesel-postgres · db-sqlx-sqlite · …
# Features are composable — enable two drivers during a migration.

// boot (plugin on_init): build pools, register per tenant, freeze
let primary = sqlx_driver(&db_url, 16).await?;
ctx.provide(DataSourceRegistry::new(ArclyDbPool::new("default", primary)));

// handler: one attribute = begin → commit on Ok / rollback on Err or timeout
#[Post("/", status(201), security("bearer"))]
#[Transactional]
#[AuditLog(action = "note.create", resource = "note")]
async fn create_note(ctx: RequestContext, #[Body] dto: CreateNoteDto) -> … {
    with_current_tx(|mut tx| async move {
        let r = async {
            tx.execute_bind("INSERT INTO notes (body) VALUES (?)", &[&dto.body]).await?;
            tx.execute_bind("INSERT INTO arcly_outbox (…) VALUES (…)", &[…]).await?;
            Ok(())                       // ← outbox rides the SAME transaction
        }.await;
        (tx, r)
    }).await?;
    …
}

Guarantees, verified by the example's /notes demo:

Path	Behaviour
Handler returns Ok	transaction commits — note and outbox row land together
Handler returns Err after both inserts	rollback — neither row survives
#[Timeout] expiry / client disconnect	future dropped → uncommitted tx → driver rollback
Sync Diesel	#[Transactional] rejects with guidance — use DieselBlockingPool::transaction(…) (whole tx in one spawn_blocking closure; workers never block)
Schema changes	MigrationRunner — ledgered, checksummed (drift fails the boot), serialized across the fleet on the cluster lock

---

Event mesh — produce transactionally, consume declaratively

// produce (inside #[Transactional] — same DB transaction as the data):
tx.execute_bind("INSERT INTO arcly_outbox (topic, payload, idempotency_key, traceparent) …").await?;

// consume (NestJS-style — registered at link time, dispatched lock-free):
#[EventConsumer]
impl NoteEventsConsumer {
    #[EventPattern("note.created")]
    async fn on_note_created(ctx: EventContext) -> Result<(), String> {
        let evt: NoteCreated = ctx.payload()?;          // typed payload
        // ctx.trace continues the producing request's trace
        ctx.inject::<SearchIndexer>().index(evt.note_id).await
    }
}

Guarantee	Mechanism
No dual writes	outbox row commits with the business data
Fleet-safe relay	leader election on the distributed lock
Effectively-once consume	claim → complete-on-success / release-on-failure via the idempotency store
Poison isolation	bounded nack-retries, then dead-letter with the failure reason
Trace continuity	traceparent column → EventContext.trace (same trace ID end-to-end)

---

PII masking — redact before durability

The compliance machinery is durable on purpose (hash-chained audit, committed outbox rows, replayable idempotency cache, DLQ) — so raw PII reaching any of them is permanent. The Masker closes that at the sink:

ctx.provide(Masker::new(MaskingPolicy::new(1)
    .field("email")                      // Redact: j***@e***.com
    .field("payment.card.number:last4")  // ************4242
    .field("ssn:drop")
    .field("items.*.patient_name:hash"), // joinable, unreadable
));

One ArcSwap load + a pure tree walk per sink write — rules hot-reload at runtime (POST /admin/masking in the example), and #[MaskFields] sits inside #[Idempotent] so replay caches only ever hold masked bodies.

---

Plugin system — extend the framework without forking it

A plugin owns a corner of the framework's behaviour through one trait. Everything it registers rides the same frozen, lock-free runtime as the core:

use arcly_http::prelude::*;
use futures::future::BoxFuture;

struct MyPlugin;

impl ArclyPlugin for MyPlugin {
    fn name(&self) -> &'static str { "my-plugin" }

    fn on_init<'a>(&'a mut self, ctx: &'a mut ArclyPluginContext)
        -> BoxFuture<'a, Result<(), PluginError>> {
        Box::pin(async move {
            ctx.provide(MyService::new());                    // new DI singleton
            ctx.override_provider(MySessionStore::new());     // replace a core provider — explicit, never silent
            ctx.add_get("/my/route", my_handler);             // mounted on the shared pipeline
            ctx.register_boundary_filter(&MY_FILTER);         // pre-body early reject (signatures, allowlists)
            ctx.register_global_interceptor(&MY_INTERCEPTOR); // wraps EVERY route, macro + plugin
            ctx.modify_openapi(|spec| { /* patch the spec */ });
            Ok(())
        })
    }
    // on_start:    listener live — spawn background tasks, mount runtime routes
    // on_draining: shutdown signal — stop consuming new work (runs WITH the HTTP drain)
    // on_shutdown: HTTP fully drained — cleanup, bounded by LaunchConfig::drain_budget per plugin
}

The lifecycle contract, enforced by App::launch_configured:

Phase	When	Guarantees
on_init	before the DI container freezes	queue providers/overrides, routes, filters, interceptors; route & provider collisions fail launch loudly (no silent winners, no axum panics)
on_start	listener is live	resolve anything via &'static FrozenDiContainer; a failure rolls back already-started plugins in reverse order
on_draining	shutdown signal received	fires concurrently with the HTTP in-flight drain — quiesce MQ consumers and schedulers
on_shutdown	HTTP fully drained	reverse order, per-plugin drain_budget timeout — a wedged plugin can't wedge the process

Two escape hatches keep "dynamic" off the hot path:

• Runtime routes — Inject<DynamicRouteTable>, then mount()/unmount() under /_plugins/* at any time. Dispatch is one ArcSwap load + one hash probe; routes outside the namespace pay nothing. The core router stays frozen.
• Request-scoped state — ctx.extensions_mut() carries typed per-request values from interceptors to handlers. The process-wide container is untouched.

See examples/enterprise_app/src/infra/shield.rs for a compact plugin that exercises this entire surface.

---

Production governance — LaunchConfig

Every operational guardrail lives in one struct, applied as outermost layers at launch (zero per-request cost for anything left disabled):

App::launch_configured::<AppModule>(addr, info, plugins,
    LaunchConfig::default()
        .request_timeout(Duration::from_secs(10))   // routes w/o #[Timeout] → 504, worker freed
        .max_in_flight(8_192)                       // beyond this → 503 + Retry-After (atomic CAS, pre-body)
        .adaptive_shed_target(Duration::from_millis(250)) // latency EWMA above this → pressure-proportional 503s
        .max_body_bytes(2 * 1024 * 1024)            // request body cap, every entry point
        .cache_max_entries(50_000)                  // #[CacheTTL] store ceiling + background sweeper
        .ws_drain_deadline(Duration::from_secs(10)) // then Close frames — shutdown can't hang on live sockets
        .drain_budget(Duration::from_secs(5))       // per-plugin on_shutdown budget
        .cors(CorsConfig::for_origins(["https://app.example.com"]))
        .expose_docs(false),                        // hide /docs + /openapi.json in hardened deployments
).await

(LaunchConfig is #[non_exhaustive] with chained setters — new knobs land without ever breaking your construction again.)

Every knob can be retuned at deploy time — ARCLY_* environment variables override the coded values (incident response without a rebuild): ARCLY_REQUEST_TIMEOUT_MS, ARCLY_MAX_IN_FLIGHT, ARCLY_ADAPTIVE_SHED_TARGET_MS, ARCLY_MAX_BODY_BYTES, ARCLY_CACHE_MAX_ENTRIES, ARCLY_WS_DRAIN_DEADLINE_MS, ARCLY_DRAIN_BUDGET_MS, ARCLY_EXPOSE_DOCS. Unparseable values are ignored with a warning, never a silent behaviour change.

Operational guarantees behind those knobs:

• Shutdown always completes: signal → /readyz flips to 503 {"status":"draining"} (LB stops routing; /healthz stays green so the supervisor doesn't kill the drain) → on_draining (concurrent with HTTP drain) → WS Close frames at the deadline → drain finishes → per-plugin on_shutdown. No state where a live socket or wedged plugin forces a SIGKILL.
• Panics become 500s: a panicking handler answers 500 with its x-request-id, increments http_handler_panics_total, and logs the payload — never a silently dropped connection no dashboard sees.
• Oversized bodies are 413, not silently truncated — a request past max_body_bytes is rejected before context assembly, so handlers never see a deceptively empty body.
• Every response carries x-request-id (inbound honoured, else minted) — including sheds, timeouts, and panics — alongside W3C traceparent.
• Probes are bounded: each health check gets a 2-second budget; a hung dependency degrades the probe instead of hanging it.
• JWT signing never panics: issue_* return Result<_, JwtSignError>; a bad rotation payload is a 500 on that request, not a dead process.
• No unbounded memory on the request path: response cache is capacity- gated and swept; /openapi.json is serialized once at boot and served as static bytes.

---

Metrics reference

Emitted via Prometheus (/metrics when the observability plugin is mounted). Everything is automatic except the first row, which rides #[UseInterceptors(TraceInterceptor)] per route/controller.

Area	Metric	Notes
HTTP	http_requests_total · http_request_duration_seconds	labelled by route pattern, method, status — via TraceInterceptor
HTTP	http_requests_in_flight (gauge) · http_requests_shed_total · http_requests_adaptive_shed_total · http_requests_deadline_total · http_handler_panics_total · http_requests_body_too_large_total	governor: admission cap, adaptive latency shedding, deadline, panic boundary, body cap
WebSocket	ws_connections (gauge) · ws_messages_in_total / ws_messages_out_total · ws_slow_client_evictions_total · ws_idle_reaped_total · ws_upgrades_refused_total · ws_handler_errors_total	bounded queues, heartbeat reaping, connection cap
Resilience	circuit_breaker_transitions_total{name,to} · rate_limited_total · bulkhead_rejected_total{name} · bulkhead_available{name} (gauge) · handler_timeouts_total{route}	breaker names = method names from #[circuit_breaker]
Database	db_acquire_seconds{pool} (histogram) · db_acquire_errors_total{pool} · db_replica_fallback_total{pool}	facade-level; replica failover visibility
Events	events_consumed_total{topic} · events_retried_total · events_dead_lettered_total · events_deduped_total · events_tenant_rejected_total	consumer mesh incl. suspended-tenant rejection
Outbox	outbox_published_total · outbox_publish_errors_total · outbox_fetch_errors_total	relay health
Idempotency	idempotency_replays_total · idempotency_conflicts_total · idempotency_store_errors_total	#[Idempotent]
Compliance	masked_responses_total · policy_denials_total · audit_dropped_total · audit_sink_errors_total	masking, ABAC, audit pipeline backpressure
Locks	dlock_backend_errors_total	distributed lock backend

---

Testing your application

use arcly_http::testing::{TestRequest, TestServer};

// Unit test a guard — no server, but the REAL boundary pipeline:
#[tokio::test]
async fn admin_guard_rejects_customers() {
    let ctx = TestRequest::get("/admin/users")
        .claims(serde_json::json!({"sub": "1", "role": "customer"}))
        .build()
        .await;
    assert!(RoleGuard("admin").check(&ctx).is_err());
}

// Integration test — full boot on an ephemeral port:
#[tokio::test]
async fn end_to_end() {
    let server = TestServer::launch::<AppModule>(vec![], LaunchConfig::default()).await;
    let resp = reqwest::get(format!("{}/healthz", server.base_url)).await.unwrap();
    assert_eq!(resp.status(), 200);
}

TestRequest supports .header(), .query(), .json(), .claims(), and .provide::<T>() for DI singletons (e.g. a TenantRegistry) — tenant resolution, trace continuation, and the body cap behave exactly as in production because it runs the same assemble_context.

---

🧰 Examples

Feature	Where
JWT + refresh rotation (Redis)	auth.rs
Cookie + session auth	auth.rs, session_store.rs
OAuth 2.0 Google/GitHub + PKCE	oauth.rs, oauth_providers.rs
resource:action permissions	permissions.rs
Module DAG (5 modules)	main.rs
Circuit-breaker on payments	services.rs
Response caching with TTL	controllers.rs (products)
WebSocket order gateway	gateways.rs
SSE inventory stream	controllers.rs
Health checks + plugin lifecycle	bootstrap.rs
Full plugin surface: boundary filter, global interceptor, runtime /_plugins route, drain hooks	shield.rs
Field encryption: dev KekSource, per-tenant DEKs, /_plugins/vault/demo	kek_source.rs, bootstrap.rs
Production governance: deadline, admission cap, CORS (LaunchConfig)	main.rs
Redis with in-memory fallback	redis.rs

WebSocket gateways, rooms, broadcast patterns, and SSE under one roof.

---

⚙️ Configuration

All settings are environment variables read at startup by the example's AppInitPlugin:

Variable	Default	Purpose
REDIS_URL	redis://127.0.0.1:6379	Token store + sessions — falls back to in-memory when unreachable
JWT_SECRET	dev default	JWT HMAC signing secret
COOKIE_SECRET	dev default	Auth-cookie HMAC secret
SESSION_SECRET	dev default	Session-cookie HMAC secret
GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET	unset	Enables the Google OAuth provider
GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET	unset	Enables the GitHub OAuth provider
OAUTH_REDIRECT_BASE	http://localhost:3000	Base URL for OAuth callbacks
ARCLY_DB_URL	sqlite:///tmp/arcly_demo.db?mode=rwc	Unified DB pool for the /notes #[Transactional] demo

Secrets (JWT_SECRET, COOKIE_SECRET, SESSION_SECRET) are watched at runtime: change the value at the source and the corresponding service hot-rotates its keys within 60 s — no restart, no mass logout. Tenants are selected per request via the X-Tenant-Id header (acme / globex, fallback public).

⚠️ Production: set real secrets, enable secure: true on cookies (TLS), and point REDIS_URL at a persistent instance — the in-memory fallback loses sessions and token state on restart.

---

🛠 Development

cargo build --workspace                   # build everything
cargo test  -p arcly-http                 # unit + doc tests
cargo test  -p arcly-http --test smoke    # integration smoke suite
cargo run   -p enterprise_app             # full example server

---

📄 License

MIT