rusty-agent-server 0.12.0

The durable agent runtime built in Rust — HTTP/SSE server: an Agent-Protocol subset with threads, background runs, checkpoints, and resumable streaming.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
# Rusty Server

**The network face of [`rusty-agent-runtime`](../rusty-core)** — serve your agent graphs over HTTP + SSE from a single static binary. No interpreter, no Postgres, no Redis. Dual-licensed under MIT OR Apache-2.0.

> **Status: v0.5, under active development.** The crate ships as a *library*: you call `rusty_agent_server::serve()` from your own `main.rs`. The endpoint set, streaming semantics, and config surface follow the architecture document in [`docs/rusty-server-design.md`](../docs/rusty-server-design.md). The core `rusty-agent-runtime` crate is untouched — it has no HTTP, no axum, no server dependencies, and never learns that a server exists.

> **New in v0.5 — the Flight Recorder surface.** Every run is journaled (core R0.5 kernel): the server attaches a journal to the executor at run start, persists its snapshot at every checkpoint boundary and at run completion (`{store_path}/journals/{run_id}.json`, or the auto-migrated `server_journals` table on Postgres), and serves it read-only via `GET /runs/{run_id}/events` — fetchable by run id even after the run's in-memory record is evicted or the process restarts. On top of that: `GET /runs/{run_id}/fixture` (portable CI replay bundle), `POST /runs/replay` (server-side exact replay with evidence verification), and `GET /runs/diff` (branch diff of two runs' journals). Fully additive — no breaking changes in this version.

## Why one binary instead of three containers

A self-hosted LangGraph Platform standalone deployment needs **three moving parts**: the API container, Postgres (threads / runs / checkpoints / task queue), and Redis (pub/sub fan-out for background-run streaming) — plus a queue-worker topology for exactly-once background runs. `rusty-agent-server` collapses that into a single static binary, because the primitives LangGraph rents from infrastructure fall out of `rusty-agent-runtime`'s execution model for free:

| Concern | LangGraph Platform | rusty-agent-server |
|---|---|---|
| User-code loading | `langgraph.json` + pip install at image build | `Cargo.toml` + `main.rs`, static link |
| Deployment unit | API image + Postgres + Redis (compose) | one static binary |
| Checkpoint store | Postgres | embedded `JsonFileCheckpointer` (wired from `ServerConfig::store_path`), or core's `PostgresCheckpointer` via `ServerConfig::with_postgres` (feature `postgres`) |
| Stream fan-out | Redis pub/sub | in-process `tokio::sync::broadcast` per run |
| Background-run queue | Postgres task queue + workers | in-process per-thread run queue |
| Stream resume | `stream_resumable` contract | replay from the per-run in-memory event log, deduped by `Last-Event-ID` |
| Multi-process scale-out | supported | Phase B gRPC worker protocol (see [roadmap](#roadmap)) |

The trade is explicit: this is a **single-process** server. That covers the overwhelming majority of self-hosted agent deployments, and the `Node` trait keeps the multi-process door open: remote gRPC workers and WASM nodes are planned implementations of the same trait, not architectural changes.

## Setup: Cargo.toml is the new langgraph.json

LangGraph's `langgraph.json` exists because Python can import user modules at runtime. Rust cannot, so in Rust, the declaration of "which graphs this server hosts" *is* your `main.rs`, and the dependency list *is* `Cargo.toml`. The server is a crate you call, not a binary you load graphs into.

```toml
[dependencies]
rusty-agent-runtime = "0.4"
rusty-agent-server = "0.5"
tokio = { version = "1", features = ["full"] }
serde_json = "1"
tracing-subscriber = "0.3"
```

A realistic `main.rs` — register a graph under a name, hand the registry to `serve`:

```rust
use std::sync::Arc;
use rusty_agent_runtime::prelude::*;
use rusty_agent_server::{serve, GraphRegistry, ServerConfig};

mod graphs; // your code: build_support_graph(), etc.

#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    // Graph 1: the prebuilt ReAct agent.
    let mut tools = ToolRegistry::new();
    tools.register(Calculator);
    tools.register(Echo);
    let model: Arc<dyn ChatModel> = Arc::new(OpenAiCompatibleClient::from_env(
        "https://api.openai.com/v1",
        "OPENAI_API_KEY",
        "gpt-4o-mini",
    ));
    let react = create_react_agent(model, tools)?;
    let react_spec = StateSpec::new().channel("messages", Reducer::AddMessages);

    // Graph 2: a custom compiled graph.
    let (support, support_spec) = graphs::build_support_graph()?;

    // The registry: the Rust analog of langgraph.json's `graphs` map.
    let mut registry = GraphRegistry::new();
    registry.register("react_agent", react, react_spec);
    registry.register("support_agent", support, support_spec);

    // One call: serve. Blocks on the axum/tokio runtime.
    let config = ServerConfig::new(
        "0.0.0.0:8080".parse()?,       // bind address
        "./data/checkpoints",          // JsonFileCheckpointer root
    );
    serve(registry, config).await?;
    Ok(())
}
```

A `GraphRegistry` entry is a name plus the two things the executor needs — a `Graph` and its `StateSpec` — so `Executor::run(&graph, &spec, state, config)` can be driven for any registered name over HTTP. Registration is **compile-checked**: a graph whose nodes write channels absent from its spec fails in your CI, not in production.

**Dev loop.** No `langgraph dev` equivalent is needed. `cargo watch -x run` (or `bacon run`) recompiles and restarts on save; incremental rebuilds of a single-graph binary take seconds. During development, point `ServerConfig::new`'s `store_path` at a scratch directory you can delete between runs.

**Embedding.** `serve(registry, config)` binds and blocks; if you want the routes inside a larger axum application (or want to drive the API in tests via `tower::ServiceExt::oneshot`), call `rusty_agent_server::router(registry, config)` instead and merge the returned `Router` yourself.

**Graceful shutdown (R0.6 wave 2c).** `serve` drains on SIGINT/SIGTERM: axum stops accepting connections and waits for in-flight requests; a shared token cooperatively cancels in-flight runs at their next super-step boundary — where a checkpoint was just persisted, so re-running the thread resumes the work — and ends them terminal-`cancelled`; new run submissions answer `503 shutting_down`; the outbox relay finishes its current pass and stops (pending rows publish on the next process's first pass); the cron scheduler stops firing. The whole drain is bounded by `ServerConfig::with_shutdown_grace` (default 25 s, under Kubernetes' 30 s pod-termination grace); past it the server stops anyway, which is the crash case the checkpoint log and lease expiry already cover. Embedders get the same pieces: `serve_with_shutdown(registry, config, future)` takes any shutdown future, `shutdown_signal()` is the SIGINT/SIGTERM default, and `router_with_shutdown(registry, config, token)` wires the cooperative drain into a self-hosted `Router`.

## HTTP API

An Agent-Protocol-compatible subset — wire-compatible with the core run/thread shapes LangGraph Platform uses, without the commercial surface. This table is the v0.5 endpoint inventory; everything listed here is implemented and covered by integration tests.

| Endpoint | Description |
|---|---|
| `GET /ok` | Liveness probe → `{"ok": true}` |
| `GET /info` | Service version, checkpointer kind, store path, registered graphs + their channels |
| `POST /threads` | Create a thread bound to a registered graph: `{graph, metadata?, thread_id?}` → `201` |
| `POST /threads/{id}/fork` | [Time travel](#time-travel-fork--checkpoint-replay): copy the thread's checkpoint history into a new thread: `{new_thread_id?, checkpoint_id?}` → `201 {thread_id, checkpoints_copied}` |
| `GET /threads/{id}/state` | Latest checkpoint: `{values, next, checkpoint}` |
| `POST /threads/{id}/state` | Write a new checkpoint (the `update_state` analog; optional `as_node`, `next_nodes`) → `201` |
| `POST /threads/{id}/history` | List checkpoints, newest first, with `limit` / `before` (`400` for an unknown `before` cursor) |
| `POST /threads/{id}/runs` | Start a **background** run → `202` + `{run_id, thread_id, status}` |
| `POST /threads/{id}/runs/wait` | Run to completion; returns the terminal JSON (`{status, output \|\| interrupt, …}`); server-side wait ceiling of 3600 s → `504` on timeout (the run keeps executing — poll `GET /runs/{id}`) |
| `POST /threads/{id}/runs/stream` | Run with [SSE streaming](#streaming-sse); a fresh run starts a new frame sequence, so `Last-Event-ID` is **ignored** here |
| `GET /runs/{id}/stream` | Attach to an existing run's SSE stream: replay the event log — **honoring `Last-Event-ID`** — then follow live frames until `end`; `404` for unknown or cross-tenant runs |
| `GET /runs/{id}/events` | [Flight Recorder](#flight-recorder-run-journals): the run's journaled `RunEvent`s → `{run_id, events, complete}`; `404` for unknown or cross-tenant runs; stays fetchable by run id after run eviction/restart via the persisted journal |
| `GET /runs/{id}/fixture` | [Flight Recorder](#flight-recorder-run-journals): download the run as a portable `ReplayFixture` bundle (journal + graph topology hash + final checkpoint) for CI replay; `409` before the first persisted snapshot |
| `POST /runs/replay` | [Flight Recorder](#flight-recorder-run-journals): re-drive a journaled run against its registered graph (zero outbound) and verify the replayed evidence → `{run_id, verified, expected_events, actual_events, first_divergence}`; `404` unknown/cross-tenant, `409` no journal or still executing, `422` graph not registered in this process / effect-carrying or resumed journal |
| `GET /runs/diff?base=&branch=` | [Flight Recorder](#flight-recorder-run-journals): structural diff of two runs' journals (core's `BranchDiff` shape); `404` unknown/cross-tenant either side, `409` when either run has no persisted journal |
| `DELETE /threads/{id}/runs/{run_id}` | Rollback: delete a **finished** run's checkpoints, re-anchoring the thread to the pre-run checkpoint (`409` while the run is active, while the thread is busy, when the run's checkpoints are no longer the history tail, or on the Postgres backend) |
| `GET /runs/{run_id}` | Poll a run: `{run_id, thread_id, graph, attempt, status}`; once terminal the body also carries the run's `output` / `error` / `interrupt` fields (up to 1024 terminal runs retained per process, oldest evicted beyond that) |
| `POST /assistants` | Create a named graph alias: `{name, graph, config?, metadata?, assistant_id?}` → `201` (persisted under `{store_path}/assistants/`) |
| `GET /assistants` / `GET /assistants/{id}` | List / fetch assistants |
| `POST /crons` | Schedule recurring runs: `{graph, interval_secs ‖ cron_expr, input?, metadata?, on_run_completed?}` → `201` (persisted under `{store_path}/crons/`) |
| `GET /crons` / `DELETE /crons/{id}` | List crons (with `runs_fired`, `last_run_at`) / delete a cron (`404` when unknown) |
| `PUT /store/{ns}/{key}` | Upsert a JSON value in a namespace → `201` on create, `200` on replace (`created_at` preserved) |
| `GET /store/{ns}/{key}` / `DELETE /store/{ns}/{key}` | Fetch / delete one item (`404` when absent) |
| `GET /store/{ns}` | List a namespace's items, sorted by key (empty array for an unwritten namespace) |

Not in v0.5 (roadmap, see below): thread listing/deletion endpoints, `/metrics`, `/graphs`, the replay-POST Flight Recorder endpoint, and the gRPC worker protocol. Thread records **are** durable: each thread persists as one JSON file under `{store_path}/threads/` (or in the `server_threads` table with [Postgres persistence](#postgres-persistence-feature-postgres)) and reloads on startup — persistence is what makes the checkpoint durability story reachable through the API, since a restart that forgot the thread records would 404 every pre-restart thread while its checkpoints sat orphaned on disk. Assistants, crons, and store items are likewise durable (JSON files under `store_path`, or the `server_*` tables) and reload on startup.

**Run-create payload** (subset of LangGraph's shape):

```json
{
  "input": { "messages": [ { "role": "user", "content": "What is 17 + 25?" } ] },
  "command": { "resume": { "approved": true } },
  "config": { "recursion_limit": 25 },
  "checkpoint": { "checkpoint_id": "optional-checkpoint-uuid" },
  "metadata": {},
  "stream_mode": ["values", "updates"],
  "multitask_strategy": "reject",
  "assistant_id": "optional-assistant-uuid"
}
```

- `assistant_id` runs through a [named assistant](#assistants-crons-and-the-kv-store): the assistant must be bound to the same graph as the thread (`400` on mismatch, `404` when unknown), and its `config.recursion_limit` applies as a default when the payload doesn't set one.
- `command.resume` is the human-in-the-loop channel: it maps directly to `RunConfig::with_resume(value)`. The executor restores the thread's latest checkpoint, re-runs the interrupted node with `NodeContext::resume_value()` returning the payload, and the run continues. An interrupted run is reported as `{"status": "interrupted", "interrupt": <value>, "checkpoint_id": …, "state": …}`.
- `checkpoint.checkpoint_id` is the time-travel channel: it maps to `RunConfig::with_checkpoint_id(id)` — the run replays from **that** checkpoint of the thread (its state and next-node set) instead of the latest. `404` when the thread has no checkpoint with that id. Prefer replaying on a [fork](#time-travel-fork--checkpoint-replay) rather than the original thread, so the new history grows on a branch instead of appending to the live timeline. Combines with `command.resume`: `checkpoint_id` selects *where* the run restarts, `resume` supplies the resume value for the first super-step.
- `config.recursion_limit` maps to `RunConfig::with_max_steps(n)`.
- `stream_mode` selects which frame families the SSE endpoint emits; default `["values", "updates"]`. `metadata`, `error`, and `end` frames are always emitted. Add `"messages"` for LLM token deltas.
- `multitask_strategy` — one active run per thread: `enqueue` (default) queues onto the per-thread run queue (depth-capped by `ServerConfig::max_concurrent_runs_per_thread`), `reject` returns `409 Conflict`. LangGraph's `rollback` strategy is instead an explicit operation: `DELETE /threads/{id}/runs/{run_id}` on a finished run.

**Client-chosen ids.** `thread_id` / `new_thread_id` / `assistant_id` / `cron_id` (including `assistant_id` in run bodies) must be non-empty, ≤ 256 chars, free of path separators (`/`, `\`), not all dots, and not one of the reserved layout names — `assistants`, `crons`, `journals`, `store`, `threads`, `latest` — which already name directories at the store root (or the `latest` pointer file inside each checkpoint dir); claiming one would write checkpoints into platform directories. Violations answer `400`. Tenant ids in `with_tenant_key` follow the same reserved-name rule with a 64-char cap, enforced at startup.

**Auth.** A single static API key checked against the `X-Api-Key` header (the LangSmith managed-deployment convention), set via `ServerConfig::with_api_key("…")`. With no key configured (the default), the server runs in dev mode with auth disabled.

**CORS.** `router()` layers `tower_http::cors::CorsLayer::permissive()` as the outermost middleware: every response carries `access-control-allow-origin: *`, and OPTIONS preflights are answered before the auth middleware runs. That makes browser clients — like the zero-build [Studio](../studio/) — work out of the box from any origin, including `file://`. **Production deployments should restrict this**: call `router()` and layer your own restrictive `CorsLayer` policy on top in your binary (allowed origins, methods, and headers narrowed to your frontend), or terminate CORS at a reverse proxy.

## Flight Recorder: run journals

Every run is journaled by the core Flight Recorder (R0.5): super-step boundaries, node inputs/outputs (with declared effect classes and measured latencies), model/tool/remote/WASM calls recorded by node code, interrupts, resumes, routing decisions, and checkpoint writes land in an append-only, hash-chained `Journal` — one per run, keyed by the server-minted run id. The server attaches that journal to the executor at run start and persists its `JournalSnapshot`:

- **at every checkpoint boundary** (flushed from the `CheckpointSaved` event path), so stored evidence trails the live journal by at most one super-step, and
- **at run completion** — success, interrupt, or error; evidence of a failed run is still evidence.

Snapshots persist as one JSON file per run under `{store_path}/journals/` (or the `server_journals` table with [Postgres persistence](#postgres-persistence-feature-postgres)) and are served read-only:

```bash
curl localhost:8080/runs/$RUN_ID/events
# {
#   "run_id": "7c1e…",
#   "events": [
#     {"id": "7c1e…:0", "run_id": "7c1e…", "thread_id": "3f2b…",
#      "node_id": null, "seq": 0, "kind": "super_step_start",
#      "effect": "pure", "input": {"kind": "inline", "value": …},
#      "output": null, "latency_ms": null, "tokens": null,
#      "cost_usd": null, "status": "ok", "parent": null,
#      "recorded_at": "2026-08-07T…Z"},
#     …
#   ],
#   "complete": true
# }
```

The `events` are core's `RunEvent`s in `seq` order, in the exact golden-pinned wire shape (`rusty-core/tests/golden/run_event.json`); event ids are deterministic (`{run_id}:{seq}`) and `parent` forms the causal chain. `complete` is `true` once the run is terminal — the served snapshot is the final journal. Guardrails: the stored snapshot's chained head hash is re-verified on every read (tampered evidence answers `500`, not a plausible-looking lie), and 404/tenant-isolation semantics are identical to `GET /runs/{id}` — a cross-tenant run is invisible here too. **Reachability:** journal reads resolve the run through the live run manager first and fall back to the persisted journal — so `/events`, `/fixture`, `/replay`, and `/diff` keep answering by run id after the run's in-memory record is evicted (past the 1024-run retention cap) or the process restarts, for as long as the store holds the journal; store-fallback reads are served as `complete` (no live writer remains). Both SDKs expose this endpoint as `run_events(run_id)` (Python) / `runEvents(runId)` (TypeScript).

**Fixture download** — `GET /runs/{run_id}/fixture` bundles the run for portable replay: the integrity-verified journal snapshot, the graph's topology hash, the run's final checkpoint, and provenance metadata, in core's `ReplayFixture` envelope (`format_version: 1`). Feed the JSON to `ReplayFixture::import` to re-drive the run in CI. A run with no persisted journal yet (queued, or before its first checkpoint) answers `409`; 404 and tenant-isolation semantics match `GET /runs/{id}`, and the served checkpoint's `thread_id` is always the external one. Server runs record under the system clock and OS entropy, so fixtures carry no logical-clock/RNG-seed parameters — byte-identical CI replay is for runs recorded with determinism seams. SDKs: `get_fixture(run_id)` / `getFixture(runId)`.

**Server-side replay** — `POST /runs/replay` with body `{"run_id": "…"}` re-drives the journaled run against the graph code registered in this process (zero outbound calls: node code re-executes, but any journaled model/tool/remote/WASM effects make the endpoint refuse — see below) over a throwaway in-memory checkpointer, and verifies the replayed evidence against the recorded journal:

```bash
curl -X POST localhost:8080/runs/replay \
  -H 'Content-Type: application/json' \
  -d '{"run_id": "'$RUN_ID'"}'
# {"run_id": "7c1e…", "verified": true, "expected_events": 12,
#  "actual_events": 12, "first_divergence": null}
```

`verified` compares the two journals on the evidence axes — kinds, nodes, sequences, effect classes, statuses, and resolved payloads — excluding per-run minted checkpoint ids and wall-clock measurements (server runs record under the system clock and OS entropy, so byte-identity is the CI-fixture story, not this one). `first_divergence` is the journal `seq` of the first disagreeing event (or of the first recorded event the replay never produced). Statuses: `404` unknown or cross-tenant run; `409` no persisted journal yet, or the run is still executing (replay verifies a final journal); `422` when the run's graph is not registered in this process, when the journal carries recorded model/tool/remote/WASM calls (server-side replay cannot serve them — download the fixture and replay it in CI), or when the run resumed from a checkpoint. SDKs: `replay_run(run_id)` / `replayRun(runId)`.

**Branch diff** — `GET /runs/diff?base=<run_id>&branch=<run_id>` returns the structural diff of two runs' journals in core's `BranchDiff` serde shape as-is: `first_divergent_seq`, the events `added` (branch) and `removed` (base) at and after the divergence point, per-super-step state-channel `step_diffs`, and token/cost `base_totals` / `branch_totals`. Events compare logically (identity and timing fields excluded), so two runs forked from one point show their shared prefix as equal; a run diffed against itself reports `first_divergent_seq: null`. `404` for an unknown or cross-tenant run on either side, `409` when either run has no persisted journal yet. SDKs: `diff_runs(base, branch)` / `diffRuns(base, branch)`.

The Studio's compare/replay UI consumes both endpoints with exactly these response shapes; the Python and TypeScript SDKs expose all four Flight Recorder endpoints (`run_events`/`get_fixture`/`replay_run`/`diff_runs`, camelCase in TypeScript).

## Time travel: fork & checkpoint replay

Every super-step boundary is a checkpoint, and every checkpoint is a handle. Two endpoints turn that into LangGraph-style time travel:

**Fork a thread's history** — `POST /threads/{id}/fork` copies the source thread's checkpoints (preserving their ids, steps, states, and next-node sets; only the `thread_id` changes) into a new thread bound to the same graph, via core's `Checkpointer::fork_thread`:

```bash
# Full-history fork
curl -X POST localhost:8080/threads/$TID/fork \
  -H 'Content-Type: application/json' -d '{}'
# -> 201 {"thread_id": "9c1e…", "checkpoints_copied": 2}

# Mid-history fork: copy only up to (and including) a checkpoint,
# so the fork branches off at that point in the timeline
curl -X POST localhost:8080/threads/$TID/fork \
  -H 'Content-Type: application/json' \
  -d '{"new_thread_id": "branch-a", "checkpoint_id": "'$CP_ID'"}'
# -> 201 {"thread_id": "branch-a", "checkpoints_copied": 1}
```

Errors: `404` when the source thread (or the `checkpoint_id`) is unknown, `400` when the source thread has no checkpoints to copy, `409` when `new_thread_id` is already taken.

**Replay a run from a checkpoint** — all three run endpoints accept `"checkpoint": {"checkpoint_id": "…"}`, which maps to `RunConfig::with_checkpoint_id(id)`: the run restores *that* checkpoint's state and next-node set instead of the latest and continues from there (`404` when the checkpoint is unknown):

```bash
# Re-run the tail of the graph from an earlier boundary, on the fork
curl -X POST localhost:8080/threads/branch-a/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"checkpoint": {"checkpoint_id": "'$CP_ID'"}}'
```

The safe pattern is **fork first, replay on the fork**: replaying on the original thread appends new history on top of the old timeline (supported, but rarely what you want), while a fork gives the alternate path its own thread id and its own history. Checkpoint ids come from `POST /threads/{id}/history`, `GET /threads/{id}/state`, or an interrupted run's `checkpoint_id` field.

## Assistants, crons, and the KV store

The v0.2 platform surface, all durable as JSON files under `store_path`:

**Assistants** bind a name plus free-form `config` / `metadata` to a registered graph, so clients can create runs by `assistant_id` instead of repeating a graph name and config. Files live at `{store_path}/assistants/{assistant_id}.json` and reload on startup.

**Crons** fire runs on a schedule. `POST /crons` takes exactly one schedule kind: `interval_secs` (fixed interval, `1..=31_536_000` s — one year; out-of-range values answer `400`) or `cron_expr` (5-field `min hour day-of-month month day-of-week`, UTC, minute resolution — parsed with the `cron` crate). A background scheduler (200 ms tick) fires each due cron by creating a **fresh thread** bound to the cron's graph and scheduling a background run with the cron's `input`. Records carry `runs_fired` / `last_run_at` bookkeeping and persist at `{store_path}/crons/{cron_id}.json`. `on_run_completed: "delete"` turns a cron into a one-shot: it removes itself once its first fired run reaches a terminal state (an in-process tombstone keeps it from re-firing while that first run drains).

**Store** is a cross-thread key-value memory: `PUT /store/{namespace}/{key}` writes any JSON value, namespaced items persist at `{store_path}/store/{namespace}/{key}.json`, and listing a namespace returns its items sorted by key. Namespace and key segments are restricted to `[A-Za-z0-9._-]` (1–128 chars) to keep the path mapping unambiguous.

## Streaming (SSE)

The executor emits one typed event stream — `GraphEvent::{SuperStep, NodeStart, NodeEnd, StateUpdate, CheckpointSaved, Token}` — and LangGraph's stream modes are **filters over that single stream**, implemented as such:

| `stream_mode` | SSE frame | Source |
|---|---|---|
| `updates` | `event: updates` — `{"step": n, "updates": {channel → post-reducer value}}` per step (the full appended list for an `Append` channel, read back from the merged state) | `GraphEvent::StateUpdate` |
| `values` | `event: values` — full state per step | the `Checkpoint.state` persisted at that step's boundary, read back from the checkpoint log |
| `messages` | `event: messages` — `{"node": …, "delta": …}` per LLM token | `GraphEvent::Token` (requires the node to stream via `ChatModel::chat_stream`) |
| `metadata` | first frame: `{run_id, thread_id, graph, attempt, metadata}` | synthesized by the server |
| `error` | `{error, message}` | `Err(RustyError)` from the executor |
| `end` | `{status: success\|\|interrupted\|\|error}` (plus `interrupt` when interrupted) | the run's `ExecutionOutcome` |

Fan-out is in-process: each run owns a `tokio::sync::broadcast` channel fed from the executor's event sink, and every attached SSE client subscribes. No Redis.

### Last-Event-ID resume

Every SSE frame carries `id: {checkpoint_id}:{step}:{seq}`, where `seq` is a per-run monotonically increasing sequence number (1-based; frames emitted before the first checkpoint use `-` as the checkpoint component). The two streaming endpoints treat `Last-Event-ID` differently, on purpose:

- `POST /threads/{id}/runs/stream` starts a **fresh run with a fresh frame sequence**, so the header is **ignored** there — applying a stale seq from a previous run would silently drop the new run's first frames.
- `GET /runs/{id}/stream` **attaches** to an existing run and **honors** the header: the server replays the run's event-log frames after the last-seen sequence number, then follows live frames until the `end` frame. This is the reconnect path: start a background run (`POST /threads/{id}/runs`), attach, and re-attach with `Last-Event-ID` after a disconnect — you skip every frame you have already seen.

The event log is a per-run, in-memory ring buffer (`ServerConfig::event_log_capacity`, default 1000 frames), so replay covers client reconnects within the server's lifetime; durable cross-restart stream resume is roadmap (checkpoints *are* the stream history, so the data to rebuild it is already on disk). Terminal runs (and their event logs) are retained for attach/polling up to 1024 runs per process; the oldest terminal runs are evicted beyond that, after which attach answers `404`.

**Proxy guidance.** Behind nginx or another reverse proxy, disable response buffering for SSE routes (`X-Accel-Buffering: no`) and flush per event, or your clients will see nothing until the buffer fills.

## Postgres persistence (feature `postgres`)

The default deployment needs no infrastructure — checkpoints, assistants, crons, and KV items are JSON files under `store_path`. When you outgrow a single process's file system (several replicas, shared state, operational tooling), build with the `postgres` feature and point the server at a database:

```toml
[dependencies]
rusty-agent-server = { version = "0.5", features = ["postgres"] }
```

```rust
let config = ServerConfig::new("0.0.0.0:8080".parse()?, "./data/checkpoints")
    .with_postgres("postgres://user:pass@localhost/rusty");
// or twelve-factor style: .with_postgres(std::env::var("DATABASE_URL")?)
```

`with_postgres(url)` switches **both** persistence layers in one call:

| Layer | Default | With `with_postgres` |
|---|---|---|
| Run checkpoints | `JsonFileCheckpointer` under `store_path` | core's `PostgresCheckpointer` → table `rusty_checkpoints` |
| Threads | `{store_path}/threads/{thread_id}.json` | table `server_threads` (record as JSONB `payload`) |
| Assistants | `{store_path}/assistants/*.json` | table `server_assistants` (record as JSONB `payload`) |
| Crons | `{store_path}/crons/*.json` | table `server_crons` (record as JSONB `payload`) |
| KV store | `{store_path}/store/{ns}/{key}.json` | table `server_kv` (`namespace` + `"key"` primary key, JSONB `value`, `created_at`/`updated_at`) |
| Run journals | `{store_path}/journals/{run_id}.json` | table `server_journals` (`run_id` primary key, JSONB snapshot, `created_at`/`updated_at`) |

All six schemas (`rusty_checkpoints` plus the five `server_*` tables) are **auto-migrated** (`CREATE TABLE IF NOT EXISTS …`) on connect; connections are established lazily on first use, so `router()` stays synchronous and the server starts even if the database is briefly unreachable (first-touch failures surface as `500`s until Postgres is back). The HTTP surface is identical either way — `GET /info` reports `"checkpointer": "postgres"` when enabled — with one deliberate exception: rollback (`DELETE /threads/{id}/runs/{run_id}`) answers `409` on the Postgres backend rather than silently deleting nothing (the `Checkpointer` trait has no delete operation, so removal goes through the JSON-file layout directly). Everything else — fork, replay, crons, thread durability across restarts, the KV store, and journal persistence — runs the same code paths against the `ServerStore` / `Checkpointer` traits.

The live-Postgres integration tests are gated and skipped by default; run them against a scratch database with:

```bash
DATABASE_URL=postgres://user:pass@localhost/rusty_test \
  cargo test --features postgres --test postgres_store -- --ignored
```

## Multi-tenancy: API keys → tenants, with full isolation

Single-key auth (`with_api_key`) covers the one-organization deployment. For a shared deployment serving several organizations, map API keys to tenants:

```rust
let config = ServerConfig::new("0.0.0.0:8080".parse()?, "./data/checkpoints")
    .with_tenant_key("acme", "sk-acme-…")     // (tenant, key) pairs
    .with_tenant_key("globex", "sk-globex-…")
    .with_api_key("sk-ops-…");                // optional: legacy key = `default` tenant
```

**Key → tenant model.** Every configured key maps to exactly one tenant. A request's `X-Api-Key` header is resolved to a tenant by the auth middleware (401 for missing/unknown keys) and everything the request touches is scoped to that tenant. With **no keys configured** the server is in open (dev) mode — no header required, all requests run as the `default` tenant, and behavior is byte-identical to pre-multi-tenancy versions.

**Isolation semantics.** Every tenant-scoped resource is namespaced per tenant: threads (including their checkpoint history and run bookkeeping), assistants, crons, and KV namespaces. Concretely, internal ids and namespaces carry a `{tenant}/` prefix at the handler layer, so the storage backends separate naturally with no schema changes:

| Resource | JSON-file layout | Postgres |
|---|---|---|
| Threads + checkpoints | records: `{store_path}/threads/{tenant}/{id}.json`; checkpoints: `{store_path}/{tenant}/{thread_id}/…` | `server_threads.thread_id` resp. `rusty_checkpoints.thread_id` = `"{tenant}/{id}"` |
| Assistants | `{store_path}/assistants/{tenant}/{id}.json` | `server_assistants.assistant_id = "{tenant}/{id}"` |
| Crons | `{store_path}/crons/{tenant}/{id}.json` | `server_crons.cron_id = "{tenant}/{id}"` |
| KV store | `{store_path}/store/{tenant}/{ns}/{key}.json` | `server_kv.namespace = "{tenant}/{ns}"` |

The `default` tenant (open mode and the legacy `with_api_key`) is **unprefixed**, so existing deployments keep their flat layout and data — upgrading is a non-event. Tenant ids must match `[A-Za-z0-9._-]` (1–64 chars).

What this buys you:

- **Same external ids can coexist across tenants.** `acme` and `globex` can both have a thread `t1`, an assistant `bot`, and a KV namespace `memories` without collisions — each resolves inside its own namespace.
- **Cross-tenant access is 404, never 403.** Fetching another tenant's thread, run, assistant, cron, or KV item returns `not_found`, exactly as if the resource didn't exist. A 403 would confirm the resource exists (and that the id is worth attacking); a 404 leaks nothing.
- **The wire never shows internal prefixes.** Responses always carry the external ids the client supplied; the `{tenant}/` prefix is an internal storage detail.
- **The cron scheduler is tenancy-aware.** It lists crons across all tenants but fires each one inside its owning tenant's namespace — cron-spawned threads and runs land in the right tenant.
- **`GET /info` stays tenant-neutral**: it exposes only the service metadata and registered graphs — no tenants, keys, or resource counts.

## Configuration

Configuration is code, via `ServerConfig` (constructed with `ServerConfig::new(bind_addr, store_path)` plus builder methods, or `ServerConfig::default()`). If you want twelve-factor env-based config in your binary, read the environment in your own `main.rs` and build the `ServerConfig` from it — the crate deliberately does not read process env itself.

| Field / builder | Default | Purpose |
|---|---|---|
| `bind_addr` | `0.0.0.0:8080` | Listen address (used by `serve`) |
| `store_path` | `./data/checkpoints` | `JsonFileCheckpointer` root (`{store_path}/{thread_id}/{checkpoint_id}.json`) and JSON-file platform persistence |
| `with_postgres(…)` | `None` = JSON files | Postgres URL for **both** checkpoints and the platform store (feature `postgres`; see [Postgres persistence](#postgres-persistence-feature-postgres)) |
| `with_api_key(…)` | `None` = dev mode, no auth | Static key required via the `X-Api-Key` header; maps to the `default` tenant |
| `with_tenant_key(…)` | — | Map an API key to a tenant for multi-tenant deployments (see [Multi-tenancy](#multi-tenancy-api-keys--tenants-with-full-isolation)) |
| `with_max_concurrent_runs_per_thread(…)` | `1` | Per-thread enqueue queue depth cap (there is always at most one *active* run per thread) |
| `with_event_log_capacity(…)` | `1000` | Per-run SSE replay buffer (frames) |

**Built-in limits** (constants, not config knobs):

| Limit | Value | Behavior |
|---|---|---|
| Terminal-run retention | 1024 runs | Run status/stream history is in-memory by design; the oldest terminal runs are evicted beyond the cap (active and queued runs are never evicted), after which `GET /runs/{id}` and `GET /runs/{id}/stream` answer `404` |
| Blocking-wait ceiling | 3600 s | `POST /threads/{id}/runs/wait` answers `504 Gateway Timeout` when the run hasn't terminated within the ceiling; the run itself keeps executing — poll `GET /runs/{id}` |
| Cron interval ceiling | 31 536 000 s (1 year) | `interval_secs` above the ceiling answers `400` (unbounded values would overflow the scheduler's timestamp math) |

## Deployment

Build one static binary:

```bash
cargo build --release
# -> target/release/my-agent   (statically linked; the bundled server_demo
#    measures ~5 MB as a macOS release build — feature sets shift the size)
```

Ship it in a scratch image — no interpreter, no pip layer, no system Python:

```dockerfile
FROM rust:1-bookworm AS build
WORKDIR /app
COPY . .
RUN cargo build --release

FROM scratch                       # or gcr.io/distroless/static
COPY --from=build /app/target/release/my-agent /my-agent
ENTRYPOINT ["/my-agent"]
```

## curl quickstart

With the server running locally in dev mode (no API key configured):

```bash
# Liveness + what's registered
curl localhost:8080/ok
# {"ok":true}
curl localhost:8080/info
# {"service":"rusty-server","version":"0.5.0","checkpointer":"json_file",
#  "store_path":"./data/checkpoints",
#  "graphs":[{"name":"react_agent","channels":["messages"]}]}

# Create a thread bound to a registered graph
curl -X POST localhost:8080/threads \
  -H 'Content-Type: application/json' \
  -d '{"graph": "react_agent"}'
# -> 201 {"thread_id": "3f2b9c…", "graph": "react_agent",
#         "metadata": null, "created_at": "2026-08-05T…Z"}

# Blocking run
curl -X POST localhost:8080/threads/$TID/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"input": {"messages": [{"role": "user", "content": "What is 17 + 25?"}]}}'

# Streaming run (SSE) — note -N to disable curl buffering
curl -N -X POST localhost:8080/threads/$TID/runs/stream \
  -H 'Content-Type: application/json' \
  -d '{"input": {"messages": [{"role": "user", "content": "Echo hi"}]},
       "stream_mode": ["updates", "values"]}'

# Resume an interrupted run (human-in-the-loop)
curl -X POST localhost:8080/threads/$TID/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"command": {"resume": {"approved": true}}}'

# Thread state + checkpoint history
curl localhost:8080/threads/$TID/state
curl -X POST localhost:8080/threads/$TID/history \
  -H 'Content-Type: application/json' -d '{"limit": 10}'

# Roll back a finished run's checkpoints
curl -X DELETE localhost:8080/threads/$TID/runs/$RUN_ID

# Time travel: fork the thread at an earlier checkpoint, replay the fork
CP_ID=$(curl -X POST localhost:8080/threads/$TID/history \
  -H 'Content-Type: application/json' -d '{"limit": 10}' \
  | jq -r '.[-1].checkpoint.checkpoint_id')
curl -X POST localhost:8080/threads/$TID/fork \
  -H 'Content-Type: application/json' \
  -d '{"checkpoint_id": "'$CP_ID'"}'
# -> 201 {"thread_id": "9c1e…", "checkpoints_copied": 1}
curl -X POST localhost:8080/threads/9c1e…/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"checkpoint": {"checkpoint_id": "'$CP_ID'"}}'

# Poll a background run's status (terminal runs carry output/error)
curl localhost:8080/runs/$RUN_ID

# Flight Recorder: the run's journaled evidence (RunEvents, seq order)
curl localhost:8080/runs/$RUN_ID/events
# …or download the run as a portable replay fixture for CI
curl localhost:8080/runs/$RUN_ID/fixture
# …or re-drive it server-side and verify the replayed evidence
curl -X POST localhost:8080/runs/replay \
  -H 'Content-Type: application/json' -d '{"run_id": "'$RUN_ID'"}'
# …or diff two runs' journals (e.g. a run vs its fork)
curl "localhost:8080/runs/diff?base=$RUN_ID&branch=$FORK_RUN_ID"

# Attach to a background run's SSE stream; reconnect with Last-Event-ID
# to skip frames you have already seen
curl -N localhost:8080/runs/$RUN_ID/stream
curl -N localhost:8080/runs/$RUN_ID/stream -H "Last-Event-ID: $LAST_FRAME_ID"

# Create an assistant and run by assistant_id
curl -X POST localhost:8080/assistants \
  -H 'Content-Type: application/json' \
  -d '{"name": "support-bot", "graph": "react_agent",
       "config": {"recursion_limit": 25}}'
curl -X POST localhost:8080/threads/$TID/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"assistant_id": "'$AID'"}'

# A cron that fires a run every 60 seconds on a fresh thread
curl -X POST localhost:8080/crons \
  -H 'Content-Type: application/json' \
  -d '{"graph": "react_agent", "interval_secs": 60,
       "input": {"messages": [{"role": "user", "content": "hourly summary"}]}}'
curl localhost:8080/crons
curl -X DELETE localhost:8080/crons/$CRON_ID

# Cross-thread KV store
curl -X PUT localhost:8080/store/memories/user-1 \
  -H 'Content-Type: application/json' -d '{"preference": "dark-mode"}'
curl localhost:8080/store/memories
curl -X DELETE localhost:8080/store/memories/user-1
```

With auth configured, add `-H "X-Api-Key: $KEY"` to every call. For a full walkthrough — project scaffolding, a two-node graph, streaming, and a complete interrupt/resume round trip — see **[docs/server-quickstart.md](../docs/server-quickstart.md)**.

## Roadmap

- [x] **Phase A — the server crate (v0.1).** `GraphRegistry`, the thread/run/SSE endpoint set, per-thread run queue with `multitask_strategy` (`enqueue` / `reject`) plus explicit rollback via `DELETE /threads/{id}/runs/{run_id}`, SSE with mode filters (`updates` / `values` / `messages`) + per-run event log + `Last-Event-ID` dedup, static API-key middleware, `JsonFileCheckpointer` wiring from `ServerConfig::store_path`. *Implemented.*
- [x] **Phase C (partial) — platform surface (v0.2).** `GET /runs/{run_id}` status polling, **assistants** (named graph + config aliases, JSON-persisted, `assistant_id` on run-create), **crons** (interval or 5-field cron schedules, durable records, background tokio scheduler firing runs on fresh threads, `on_run_completed: keep|delete`), and the cross-thread **KV store** (`/store/{namespace}/{key}`, JSON-file-backed). *Implemented.*
- [x] **Phase C (continued) — time travel + Postgres persistence (v0.3).** `POST /threads/{id}/fork` (full- or mid-history forks via core's `Checkpointer::fork_thread`), checkpoint replay on all run endpoints (`"checkpoint": {"checkpoint_id": …}` → `RunConfig::with_checkpoint_id`), and the `postgres` feature: `ServerConfig::with_postgres(url)` switches the run checkpointer to `PostgresCheckpointer` and the assistants/crons/KV surface to auto-migrated `server_assistants` / `server_crons` / `server_kv` tables behind a `ServerStore` trait. *Implemented.*
- [x] **Phase C (continued) — permissive CORS (v0.3).** `router()` layers `tower_http::cors::CorsLayer::permissive()`, so browser clients (the [Studio](../studio/)) call the API cross-origin; preflights are answered before auth. Restrict for production — see [CORS](#http-api). *Implemented.*
- [x] **Phase C (continued) — multi-tenancy (v0.4).** API keys map to tenants (`with_tenant_key(tenant, key)`; legacy `with_api_key` = the `default` tenant); threads + checkpoints, runs, assistants, crons, and KV namespaces are fully isolated via internal `{tenant}/` id prefixing, with cross-tenant access answering 404 (never 403). Open mode and the default tenant keep the legacy flat storage layout. See [Multi-tenancy](#multi-tenancy-api-keys--tenants-with-full-isolation). *Implemented.*
- [x] **Phase C (continued) — hardening (v0.4).** Durable thread records (`threads/` JSON files / `server_threads` table — pre-restart checkpoints stay reachable through the API), the SSE attach endpoint (`GET /runs/{id}/stream` with `Last-Event-ID` replay), rollback guards (`409` on the Postgres backend, on busy threads, and on mid-history suffix violations), the cron `interval_secs` clamp (≤ 1 year) + one-shot tombstones, reserved layout names rejected as client-chosen ids, the 1024-run retention cap, the 3600 s blocking-wait ceiling (`504`), and `400` for unknown history `before` cursors. *Implemented.*
- [x] **R0.5 — Flight Recorder read surface (v0.5).** Every run is journaled by the executor (core R0.5 kernel); the server persists the journal snapshot at every checkpoint boundary and at run completion (`{store_path}/journals/{run_id}.json`, or the auto-migrated `server_journals` table on Postgres) and serves it via `GET /runs/{run_id}/events` → `{run_id, events, complete}` in the golden-pinned `RunEvent` wire shape, with head-hash re-verification on read and 404/tenant-isolation semantics identical to `GET /runs/{id}`. `GET /runs/{run_id}/fixture` downloads the run as a portable `ReplayFixture` for CI replay. SDK parity: `run_events(run_id)` (Python) / `runEvents(runId)` (TypeScript). *Implemented — the replay-POST endpoint lands in a later R0.5 wave.*
- [ ] **Phase B — gRPC worker protocol (`rusty-proto`).** `RemoteNode`: a gRPC client behind the same `Node` trait, delegating node execution to stateless out-of-process workers that long-poll named node-queues. The server keeps checkpoints, super-step scheduling, interrupts, and stream fan-out. Agent nodes are dominated by LLM latency (hundreds of ms to minutes), so a 1–5 ms gRPC hop is <1% overhead — and since `State` is already a JSON map, the wire boundary is lossless. Crash isolation, polyglot workers (a Python worker can host the LangChain ecosystem while Rust owns orchestration), and independent scaling of tool-heavy nodes follow.
- [ ] **Phase C (remainder).** Thread listing/deletion endpoints, `/metrics`, and `/graphs`. (`WasmNode` is implemented in core `rusty-agent-runtime` v0.4 behind the `wasm` feature — register a Wasm-backed graph and this crate serves it unchanged.)

Deliberately skipped: A2A/MCP server endpoints and WebSocket "protocol v2" (SSE + HTTP sidecar is sufficient), and `feedback_keys` (LangSmith-tracing coupling we don't have).

## License

Dual-licensed under either of

- Apache License, Version 2.0 ([LICENSE-APACHE](../LICENSE-APACHE))
- MIT License ([LICENSE-MIT](../LICENSE-MIT))

at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work by you, as defined in the Apache-2.0 license, shall be dual-licensed as above, without any additional terms or conditions.