# Operations guide
## Required production components
`delegated` is a library, not a service. A production deployment must supply:
1. Secure issuer signing-key custody and a rotation procedure.
2. Trusted distribution of issuer verification keys.
3. Shared revocation, emergency-deny, and atomic nonce storage.
Use [`delegated-redis`](../crates/delegated-redis/OPERATIONS.md) for Redis-backed
multi-instance deployments.
4. Durable audit persistence.
5. TLS and bearer-token handling controls.
6. Host logic that derives `OperationContext` from the operation being executed.
## Key rotation
- Publish the new `(issuer, key_id)` verification key before issuing tokens with it.
- Continue resolving the previous key until every token it signed has expired.
- Remove a compromised key immediately and revoke affected token identifiers where known.
- Never reuse a `key_id` for different key material unless replacement is an intentional emergency
action and caches are controlled.
`PinnedIssuerKeys::insert` replaces an existing mapping. Coordinate mutation so concurrent
evaluators observe an intentional key set.
## Replay state
`consume_nonce` is the critical distributed operation. It must atomically perform
"insert if absent" in the `(issuer, nonce)` namespace and retain the entry through token expiry.
Eventual consistency without a single atomic authority can permit concurrent replay.
The in-memory implementation prunes expired nonces during successful consumption and loses all
state on process restart.
## Audit behavior
`Evaluator::evaluate` returns an event but does not persist it. `evaluate_and_audit` writes through
an `AuditSink` and returns an error instead of a decision if persistence fails.
Nonce consumption occurs before audit persistence. If an audit write fails, the operation must not
execute and the token will usually be rejected as replay on retry. This favors security over
availability.
For denied requests, `claims_verified = false`; identity-like fields may be copied from untrusted
input only for diagnostics. Do not aggregate them as authenticated identities.
## Monitoring
Instrument the host around evaluation and monitor:
- allows and denies by stable `stage`;
- issuer-key resolver errors and unknown key identifiers;
- trust-state errors and replay denials;
- audit persistence errors;
- evaluation latency and backend latency;
- token lifetime and clock-skew rejection rates.
## Incident actions
- **Compromised token:** revoke `(issuer, token_id)` and inspect audit history.
- **Compromised agent:** emergency-deny `(issuer, agent_id)`.
- **Compromised issuer key:** remove the trusted key, rotate, and treat all unexpired tokens signed
by it as compromised.
- **State outage:** restore the backend; do not bypass fail-closed behavior.
- **Clock incident:** repair time synchronization; do not compensate with excessive leeway.
The evaluator caps clock leeway at five minutes. The default is 30 seconds and the default maximum
token lifetime is 24 hours.