rusty-agent-server 0.12.0

The durable agent runtime built in Rust — HTTP/SSE server: an Agent-Protocol subset with threads, background runs, checkpoints, and resumable streaming.
Documentation

Rusty Server

The network face of rusty-agent-runtime — serve your agent graphs over HTTP + SSE from a single static binary. No interpreter, no Postgres, no Redis. Dual-licensed under MIT OR Apache-2.0.

Status: v0.5, under active development. The crate ships as a library: you call rusty_agent_server::serve() from your own main.rs. The endpoint set, streaming semantics, and config surface follow the architecture document in docs/rusty-server-design.md. The core rusty-agent-runtime crate is untouched — it has no HTTP, no axum, no server dependencies, and never learns that a server exists.

New in v0.5 — the Flight Recorder surface. Every run is journaled (core R0.5 kernel): the server attaches a journal to the executor at run start, persists its snapshot at every checkpoint boundary and at run completion ({store_path}/journals/{run_id}.json, or the auto-migrated server_journals table on Postgres), and serves it read-only via GET /runs/{run_id}/events — fetchable by run id even after the run's in-memory record is evicted or the process restarts. On top of that: GET /runs/{run_id}/fixture (portable CI replay bundle), POST /runs/replay (server-side exact replay with evidence verification), and GET /runs/diff (branch diff of two runs' journals). Fully additive — no breaking changes in this version.

Why one binary instead of three containers

A self-hosted LangGraph Platform standalone deployment needs three moving parts: the API container, Postgres (threads / runs / checkpoints / task queue), and Redis (pub/sub fan-out for background-run streaming) — plus a queue-worker topology for exactly-once background runs. rusty-agent-server collapses that into a single static binary, because the primitives LangGraph rents from infrastructure fall out of rusty-agent-runtime's execution model for free:

Concern LangGraph Platform rusty-agent-server
User-code loading langgraph.json + pip install at image build Cargo.toml + main.rs, static link
Deployment unit API image + Postgres + Redis (compose) one static binary
Checkpoint store Postgres embedded JsonFileCheckpointer (wired from ServerConfig::store_path), or core's PostgresCheckpointer via ServerConfig::with_postgres (feature postgres)
Stream fan-out Redis pub/sub in-process tokio::sync::broadcast per run
Background-run queue Postgres task queue + workers in-process per-thread run queue
Stream resume stream_resumable contract replay from the per-run in-memory event log, deduped by Last-Event-ID
Multi-process scale-out supported Phase B gRPC worker protocol (see roadmap)

The trade is explicit: this is a single-process server. That covers the overwhelming majority of self-hosted agent deployments, and the Node trait keeps the multi-process door open: remote gRPC workers and WASM nodes are planned implementations of the same trait, not architectural changes.

Setup: Cargo.toml is the new langgraph.json

LangGraph's langgraph.json exists because Python can import user modules at runtime. Rust cannot, so in Rust, the declaration of "which graphs this server hosts" is your main.rs, and the dependency list is Cargo.toml. The server is a crate you call, not a binary you load graphs into.

[dependencies]
rusty-agent-runtime = "0.4"
rusty-agent-server = "0.5"
tokio = { version = "1", features = ["full"] }
serde_json = "1"
tracing-subscriber = "0.3"

A realistic main.rs — register a graph under a name, hand the registry to serve:

use std::sync::Arc;
use rusty_agent_runtime::prelude::*;
use rusty_agent_server::{serve, GraphRegistry, ServerConfig};

mod graphs; // your code: build_support_graph(), etc.

#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    // Graph 1: the prebuilt ReAct agent.
    let mut tools = ToolRegistry::new();
    tools.register(Calculator);
    tools.register(Echo);
    let model: Arc<dyn ChatModel> = Arc::new(OpenAiCompatibleClient::from_env(
        "https://api.openai.com/v1",
        "OPENAI_API_KEY",
        "gpt-4o-mini",
    ));
    let react = create_react_agent(model, tools)?;
    let react_spec = StateSpec::new().channel("messages", Reducer::AddMessages);

    // Graph 2: a custom compiled graph.
    let (support, support_spec) = graphs::build_support_graph()?;

    // The registry: the Rust analog of langgraph.json's `graphs` map.
    let mut registry = GraphRegistry::new();
    registry.register("react_agent", react, react_spec);
    registry.register("support_agent", support, support_spec);

    // One call: serve. Blocks on the axum/tokio runtime.
    let config = ServerConfig::new(
        "0.0.0.0:8080".parse()?,       // bind address
        "./data/checkpoints",          // JsonFileCheckpointer root
    );
    serve(registry, config).await?;
    Ok(())
}

A GraphRegistry entry is a name plus the two things the executor needs — a Graph and its StateSpec — so Executor::run(&graph, &spec, state, config) can be driven for any registered name over HTTP. Registration is compile-checked: a graph whose nodes write channels absent from its spec fails in your CI, not in production.

Dev loop. No langgraph dev equivalent is needed. cargo watch -x run (or bacon run) recompiles and restarts on save; incremental rebuilds of a single-graph binary take seconds. During development, point ServerConfig::new's store_path at a scratch directory you can delete between runs.

Embedding. serve(registry, config) binds and blocks; if you want the routes inside a larger axum application (or want to drive the API in tests via tower::ServiceExt::oneshot), call rusty_agent_server::router(registry, config) instead and merge the returned Router yourself.

Graceful shutdown (R0.6 wave 2c). serve drains on SIGINT/SIGTERM: axum stops accepting connections and waits for in-flight requests; a shared token cooperatively cancels in-flight runs at their next super-step boundary — where a checkpoint was just persisted, so re-running the thread resumes the work — and ends them terminal-cancelled; new run submissions answer 503 shutting_down; the outbox relay finishes its current pass and stops (pending rows publish on the next process's first pass); the cron scheduler stops firing. The whole drain is bounded by ServerConfig::with_shutdown_grace (default 25 s, under Kubernetes' 30 s pod-termination grace); past it the server stops anyway, which is the crash case the checkpoint log and lease expiry already cover. Embedders get the same pieces: serve_with_shutdown(registry, config, future) takes any shutdown future, shutdown_signal() is the SIGINT/SIGTERM default, and router_with_shutdown(registry, config, token) wires the cooperative drain into a self-hosted Router.

HTTP API

An Agent-Protocol-compatible subset — wire-compatible with the core run/thread shapes LangGraph Platform uses, without the commercial surface. This table is the v0.5 endpoint inventory; everything listed here is implemented and covered by integration tests.

Endpoint Description
GET /ok Liveness probe → {"ok": true}
GET /info Service version, checkpointer kind, store path, registered graphs + their channels
POST /threads Create a thread bound to a registered graph: {graph, metadata?, thread_id?} → 201
POST /threads/{id}/fork Time travel: copy the thread's checkpoint history into a new thread: {new_thread_id?, checkpoint_id?} → 201 {thread_id, checkpoints_copied}
GET /threads/{id}/state Latest checkpoint: {values, next, checkpoint}
POST /threads/{id}/state Write a new checkpoint (the update_state analog; optional as_node, next_nodes) → 201
POST /threads/{id}/history List checkpoints, newest first, with limit / before (400 for an unknown before cursor)
POST /threads/{id}/runs Start a background run → 202 + {run_id, thread_id, status}
POST /threads/{id}/runs/wait Run to completion; returns the terminal JSON ({status, output || interrupt, …}); server-side wait ceiling of 3600 s → 504 on timeout (the run keeps executing — poll GET /runs/{id})
POST /threads/{id}/runs/stream Run with SSE streaming; a fresh run starts a new frame sequence, so Last-Event-ID is ignored here
GET /runs/{id}/stream Attach to an existing run's SSE stream: replay the event log — honoring Last-Event-ID — then follow live frames until end; 404 for unknown or cross-tenant runs
GET /runs/{id}/events Flight Recorder: the run's journaled RunEvents → {run_id, events, complete}; 404 for unknown or cross-tenant runs; stays fetchable by run id after run eviction/restart via the persisted journal
GET /runs/{id}/fixture Flight Recorder: download the run as a portable ReplayFixture bundle (journal + graph topology hash + final checkpoint) for CI replay; 409 before the first persisted snapshot
POST /runs/replay Flight Recorder: re-drive a journaled run against its registered graph (zero outbound) and verify the replayed evidence → {run_id, verified, expected_events, actual_events, first_divergence}; 404 unknown/cross-tenant, 409 no journal or still executing, 422 graph not registered in this process / effect-carrying or resumed journal
GET /runs/diff?base=&branch= Flight Recorder: structural diff of two runs' journals (core's BranchDiff shape); 404 unknown/cross-tenant either side, 409 when either run has no persisted journal
DELETE /threads/{id}/runs/{run_id} Rollback: delete a finished run's checkpoints, re-anchoring the thread to the pre-run checkpoint (409 while the run is active, while the thread is busy, when the run's checkpoints are no longer the history tail, or on the Postgres backend)
GET /runs/{run_id} Poll a run: {run_id, thread_id, graph, attempt, status}; once terminal the body also carries the run's output / error / interrupt fields (up to 1024 terminal runs retained per process, oldest evicted beyond that)
POST /assistants Create a named graph alias: {name, graph, config?, metadata?, assistant_id?} → 201 (persisted under {store_path}/assistants/)
GET /assistants / GET /assistants/{id} List / fetch assistants
POST /crons Schedule recurring runs: {graph, interval_secs ‖ cron_expr, input?, metadata?, on_run_completed?} → 201 (persisted under {store_path}/crons/)
GET /crons / DELETE /crons/{id} List crons (with runs_fired, last_run_at) / delete a cron (404 when unknown)
PUT /store/{ns}/{key} Upsert a JSON value in a namespace → 201 on create, 200 on replace (created_at preserved)
GET /store/{ns}/{key} / DELETE /store/{ns}/{key} Fetch / delete one item (404 when absent)
GET /store/{ns} List a namespace's items, sorted by key (empty array for an unwritten namespace)

Not in v0.5 (roadmap, see below): thread listing/deletion endpoints, /metrics, /graphs, the replay-POST Flight Recorder endpoint, and the gRPC worker protocol. Thread records are durable: each thread persists as one JSON file under {store_path}/threads/ (or in the server_threads table with Postgres persistence) and reloads on startup — persistence is what makes the checkpoint durability story reachable through the API, since a restart that forgot the thread records would 404 every pre-restart thread while its checkpoints sat orphaned on disk. Assistants, crons, and store items are likewise durable (JSON files under store_path, or the server_* tables) and reload on startup.

Run-create payload (subset of LangGraph's shape):

{
  "input": { "messages": [ { "role": "user", "content": "What is 17 + 25?" } ] },
  "command": { "resume": { "approved": true } },
  "config": { "recursion_limit": 25 },
  "checkpoint": { "checkpoint_id": "optional-checkpoint-uuid" },
  "metadata": {},
  "stream_mode": ["values", "updates"],
  "multitask_strategy": "reject",
  "assistant_id": "optional-assistant-uuid"
}
  • assistant_id runs through a named assistant: the assistant must be bound to the same graph as the thread (400 on mismatch, 404 when unknown), and its config.recursion_limit applies as a default when the payload doesn't set one.
  • command.resume is the human-in-the-loop channel: it maps directly to RunConfig::with_resume(value). The executor restores the thread's latest checkpoint, re-runs the interrupted node with NodeContext::resume_value() returning the payload, and the run continues. An interrupted run is reported as {"status": "interrupted", "interrupt": <value>, "checkpoint_id": …, "state": …}.
  • checkpoint.checkpoint_id is the time-travel channel: it maps to RunConfig::with_checkpoint_id(id) — the run replays from that checkpoint of the thread (its state and next-node set) instead of the latest. 404 when the thread has no checkpoint with that id. Prefer replaying on a fork rather than the original thread, so the new history grows on a branch instead of appending to the live timeline. Combines with command.resume: checkpoint_id selects where the run restarts, resume supplies the resume value for the first super-step.
  • config.recursion_limit maps to RunConfig::with_max_steps(n).
  • stream_mode selects which frame families the SSE endpoint emits; default ["values", "updates"]. metadata, error, and end frames are always emitted. Add "messages" for LLM token deltas.
  • multitask_strategy — one active run per thread: enqueue (default) queues onto the per-thread run queue (depth-capped by ServerConfig::max_concurrent_runs_per_thread), reject returns 409 Conflict. LangGraph's rollback strategy is instead an explicit operation: DELETE /threads/{id}/runs/{run_id} on a finished run.

Client-chosen ids. thread_id / new_thread_id / assistant_id / cron_id (including assistant_id in run bodies) must be non-empty, ≤ 256 chars, free of path separators (/, \), not all dots, and not one of the reserved layout names — assistants, crons, journals, store, threads, latest — which already name directories at the store root (or the latest pointer file inside each checkpoint dir); claiming one would write checkpoints into platform directories. Violations answer 400. Tenant ids in with_tenant_key follow the same reserved-name rule with a 64-char cap, enforced at startup.

Auth. A single static API key checked against the X-Api-Key header (the LangSmith managed-deployment convention), set via ServerConfig::with_api_key("…"). With no key configured (the default), the server runs in dev mode with auth disabled.

CORS. router() layers tower_http::cors::CorsLayer::permissive() as the outermost middleware: every response carries access-control-allow-origin: *, and OPTIONS preflights are answered before the auth middleware runs. That makes browser clients — like the zero-build Studio — work out of the box from any origin, including file://. Production deployments should restrict this: call router() and layer your own restrictive CorsLayer policy on top in your binary (allowed origins, methods, and headers narrowed to your frontend), or terminate CORS at a reverse proxy.

Flight Recorder: run journals

Every run is journaled by the core Flight Recorder (R0.5): super-step boundaries, node inputs/outputs (with declared effect classes and measured latencies), model/tool/remote/WASM calls recorded by node code, interrupts, resumes, routing decisions, and checkpoint writes land in an append-only, hash-chained Journal — one per run, keyed by the server-minted run id. The server attaches that journal to the executor at run start and persists its JournalSnapshot:

  • at every checkpoint boundary (flushed from the CheckpointSaved event path), so stored evidence trails the live journal by at most one super-step, and
  • at run completion — success, interrupt, or error; evidence of a failed run is still evidence.

Snapshots persist as one JSON file per run under {store_path}/journals/ (or the server_journals table with Postgres persistence) and are served read-only:

curl localhost:8080/runs/$RUN_ID/events
# {
#   "run_id": "7c1e…",
#   "events": [
#     {"id": "7c1e…:0", "run_id": "7c1e…", "thread_id": "3f2b…",
#      "node_id": null, "seq": 0, "kind": "super_step_start",
#      "effect": "pure", "input": {"kind": "inline", "value": …},
#      "output": null, "latency_ms": null, "tokens": null,
#      "cost_usd": null, "status": "ok", "parent": null,
#      "recorded_at": "2026-08-07T…Z"},
#     …
#   ],
#   "complete": true
# }

The events are core's RunEvents in seq order, in the exact golden-pinned wire shape (rusty-core/tests/golden/run_event.json); event ids are deterministic ({run_id}:{seq}) and parent forms the causal chain. complete is true once the run is terminal — the served snapshot is the final journal. Guardrails: the stored snapshot's chained head hash is re-verified on every read (tampered evidence answers 500, not a plausible-looking lie), and 404/tenant-isolation semantics are identical to GET /runs/{id} — a cross-tenant run is invisible here too. Reachability: journal reads resolve the run through the live run manager first and fall back to the persisted journal — so /events, /fixture, /replay, and /diff keep answering by run id after the run's in-memory record is evicted (past the 1024-run retention cap) or the process restarts, for as long as the store holds the journal; store-fallback reads are served as complete (no live writer remains). Both SDKs expose this endpoint as run_events(run_id) (Python) / runEvents(runId) (TypeScript).

Fixture download — GET /runs/{run_id}/fixture bundles the run for portable replay: the integrity-verified journal snapshot, the graph's topology hash, the run's final checkpoint, and provenance metadata, in core's ReplayFixture envelope (format_version: 1). Feed the JSON to ReplayFixture::import to re-drive the run in CI. A run with no persisted journal yet (queued, or before its first checkpoint) answers 409; 404 and tenant-isolation semantics match GET /runs/{id}, and the served checkpoint's thread_id is always the external one. Server runs record under the system clock and OS entropy, so fixtures carry no logical-clock/RNG-seed parameters — byte-identical CI replay is for runs recorded with determinism seams. SDKs: get_fixture(run_id) / getFixture(runId).

Server-side replay — POST /runs/replay with body {"run_id": "…"} re-drives the journaled run against the graph code registered in this process (zero outbound calls: node code re-executes, but any journaled model/tool/remote/WASM effects make the endpoint refuse — see below) over a throwaway in-memory checkpointer, and verifies the replayed evidence against the recorded journal:

curl -X POST localhost:8080/runs/replay \
  -H 'Content-Type: application/json' \
  -d '{"run_id": "'$RUN_ID'"}'
# {"run_id": "7c1e…", "verified": true, "expected_events": 12,
#  "actual_events": 12, "first_divergence": null}

verified compares the two journals on the evidence axes — kinds, nodes, sequences, effect classes, statuses, and resolved payloads — excluding per-run minted checkpoint ids and wall-clock measurements (server runs record under the system clock and OS entropy, so byte-identity is the CI-fixture story, not this one). first_divergence is the journal seq of the first disagreeing event (or of the first recorded event the replay never produced). Statuses: 404 unknown or cross-tenant run; 409 no persisted journal yet, or the run is still executing (replay verifies a final journal); 422 when the run's graph is not registered in this process, when the journal carries recorded model/tool/remote/WASM calls (server-side replay cannot serve them — download the fixture and replay it in CI), or when the run resumed from a checkpoint. SDKs: replay_run(run_id) / replayRun(runId).

Branch diff — GET /runs/diff?base=<run_id>&branch=<run_id> returns the structural diff of two runs' journals in core's BranchDiff serde shape as-is: first_divergent_seq, the events added (branch) and removed (base) at and after the divergence point, per-super-step state-channel step_diffs, and token/cost base_totals / branch_totals. Events compare logically (identity and timing fields excluded), so two runs forked from one point show their shared prefix as equal; a run diffed against itself reports first_divergent_seq: null. 404 for an unknown or cross-tenant run on either side, 409 when either run has no persisted journal yet. SDKs: diff_runs(base, branch) / diffRuns(base, branch).

The Studio's compare/replay UI consumes both endpoints with exactly these response shapes; the Python and TypeScript SDKs expose all four Flight Recorder endpoints (run_events/get_fixture/replay_run/diff_runs, camelCase in TypeScript).

Time travel: fork & checkpoint replay

Every super-step boundary is a checkpoint, and every checkpoint is a handle. Two endpoints turn that into LangGraph-style time travel:

Fork a thread's history — POST /threads/{id}/fork copies the source thread's checkpoints (preserving their ids, steps, states, and next-node sets; only the thread_id changes) into a new thread bound to the same graph, via core's Checkpointer::fork_thread:

# Full-history fork
curl -X POST localhost:8080/threads/$TID/fork \
  -H 'Content-Type: application/json' -d '{}'
# -> 201 {"thread_id": "9c1e…", "checkpoints_copied": 2}

# Mid-history fork: copy only up to (and including) a checkpoint,
# so the fork branches off at that point in the timeline
curl -X POST localhost:8080/threads/$TID/fork \
  -H 'Content-Type: application/json' \
  -d '{"new_thread_id": "branch-a", "checkpoint_id": "'$CP_ID'"}'
# -> 201 {"thread_id": "branch-a", "checkpoints_copied": 1}

Errors: 404 when the source thread (or the checkpoint_id) is unknown, 400 when the source thread has no checkpoints to copy, 409 when new_thread_id is already taken.

Replay a run from a checkpoint — all three run endpoints accept "checkpoint": {"checkpoint_id": "…"}, which maps to RunConfig::with_checkpoint_id(id): the run restores that checkpoint's state and next-node set instead of the latest and continues from there (404 when the checkpoint is unknown):

# Re-run the tail of the graph from an earlier boundary, on the fork
curl -X POST localhost:8080/threads/branch-a/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"checkpoint": {"checkpoint_id": "'$CP_ID'"}}'

The safe pattern is fork first, replay on the fork: replaying on the original thread appends new history on top of the old timeline (supported, but rarely what you want), while a fork gives the alternate path its own thread id and its own history. Checkpoint ids come from POST /threads/{id}/history, GET /threads/{id}/state, or an interrupted run's checkpoint_id field.

Assistants, crons, and the KV store

The v0.2 platform surface, all durable as JSON files under store_path:

Assistants bind a name plus free-form config / metadata to a registered graph, so clients can create runs by assistant_id instead of repeating a graph name and config. Files live at {store_path}/assistants/{assistant_id}.json and reload on startup.

Crons fire runs on a schedule. POST /crons takes exactly one schedule kind: interval_secs (fixed interval, 1..=31_536_000 s — one year; out-of-range values answer 400) or cron_expr (5-field min hour day-of-month month day-of-week, UTC, minute resolution — parsed with the cron crate). A background scheduler (200 ms tick) fires each due cron by creating a fresh thread bound to the cron's graph and scheduling a background run with the cron's input. Records carry runs_fired / last_run_at bookkeeping and persist at {store_path}/crons/{cron_id}.json. on_run_completed: "delete" turns a cron into a one-shot: it removes itself once its first fired run reaches a terminal state (an in-process tombstone keeps it from re-firing while that first run drains).

Store is a cross-thread key-value memory: PUT /store/{namespace}/{key} writes any JSON value, namespaced items persist at {store_path}/store/{namespace}/{key}.json, and listing a namespace returns its items sorted by key. Namespace and key segments are restricted to [A-Za-z0-9._-] (1–128 chars) to keep the path mapping unambiguous.

Streaming (SSE)

The executor emits one typed event stream — GraphEvent::{SuperStep, NodeStart, NodeEnd, StateUpdate, CheckpointSaved, Token} — and LangGraph's stream modes are filters over that single stream, implemented as such:

stream_mode SSE frame Source
updates event: updates — {"step": n, "updates": {channel → post-reducer value}} per step (the full appended list for an Append channel, read back from the merged state) GraphEvent::StateUpdate
values event: values — full state per step the Checkpoint.state persisted at that step's boundary, read back from the checkpoint log
messages event: messages — {"node": …, "delta": …} per LLM token GraphEvent::Token (requires the node to stream via ChatModel::chat_stream)
metadata first frame: {run_id, thread_id, graph, attempt, metadata} synthesized by the server
error {error, message} Err(RustyError) from the executor
end {status: success||interrupted||error} (plus interrupt when interrupted) the run's ExecutionOutcome

Fan-out is in-process: each run owns a tokio::sync::broadcast channel fed from the executor's event sink, and every attached SSE client subscribes. No Redis.

Last-Event-ID resume

Every SSE frame carries id: {checkpoint_id}:{step}:{seq}, where seq is a per-run monotonically increasing sequence number (1-based; frames emitted before the first checkpoint use - as the checkpoint component). The two streaming endpoints treat Last-Event-ID differently, on purpose:

  • POST /threads/{id}/runs/stream starts a fresh run with a fresh frame sequence, so the header is ignored there — applying a stale seq from a previous run would silently drop the new run's first frames.
  • GET /runs/{id}/stream attaches to an existing run and honors the header: the server replays the run's event-log frames after the last-seen sequence number, then follows live frames until the end frame. This is the reconnect path: start a background run (POST /threads/{id}/runs), attach, and re-attach with Last-Event-ID after a disconnect — you skip every frame you have already seen.

The event log is a per-run, in-memory ring buffer (ServerConfig::event_log_capacity, default 1000 frames), so replay covers client reconnects within the server's lifetime; durable cross-restart stream resume is roadmap (checkpoints are the stream history, so the data to rebuild it is already on disk). Terminal runs (and their event logs) are retained for attach/polling up to 1024 runs per process; the oldest terminal runs are evicted beyond that, after which attach answers 404.

Proxy guidance. Behind nginx or another reverse proxy, disable response buffering for SSE routes (X-Accel-Buffering: no) and flush per event, or your clients will see nothing until the buffer fills.

Postgres persistence (feature postgres)

The default deployment needs no infrastructure — checkpoints, assistants, crons, and KV items are JSON files under store_path. When you outgrow a single process's file system (several replicas, shared state, operational tooling), build with the postgres feature and point the server at a database:

[dependencies]
rusty-agent-server = { version = "0.5", features = ["postgres"] }
let config = ServerConfig::new("0.0.0.0:8080".parse()?, "./data/checkpoints")
    .with_postgres("postgres://user:pass@localhost/rusty");
// or twelve-factor style: .with_postgres(std::env::var("DATABASE_URL")?)

with_postgres(url) switches both persistence layers in one call:

Layer Default With with_postgres
Run checkpoints JsonFileCheckpointer under store_path core's PostgresCheckpointer → table rusty_checkpoints
Threads {store_path}/threads/{thread_id}.json table server_threads (record as JSONB payload)
Assistants {store_path}/assistants/*.json table server_assistants (record as JSONB payload)
Crons {store_path}/crons/*.json table server_crons (record as JSONB payload)
KV store {store_path}/store/{ns}/{key}.json table server_kv (namespace + "key" primary key, JSONB value, created_at/updated_at)
Run journals {store_path}/journals/{run_id}.json table server_journals (run_id primary key, JSONB snapshot, created_at/updated_at)

All six schemas (rusty_checkpoints plus the five server_* tables) are auto-migrated (CREATE TABLE IF NOT EXISTS …) on connect; connections are established lazily on first use, so router() stays synchronous and the server starts even if the database is briefly unreachable (first-touch failures surface as 500s until Postgres is back). The HTTP surface is identical either way — GET /info reports "checkpointer": "postgres" when enabled — with one deliberate exception: rollback (DELETE /threads/{id}/runs/{run_id}) answers 409 on the Postgres backend rather than silently deleting nothing (the Checkpointer trait has no delete operation, so removal goes through the JSON-file layout directly). Everything else — fork, replay, crons, thread durability across restarts, the KV store, and journal persistence — runs the same code paths against the ServerStore / Checkpointer traits.

The live-Postgres integration tests are gated and skipped by default; run them against a scratch database with:

DATABASE_URL=postgres://user:pass@localhost/rusty_test \
  cargo test --features postgres --test postgres_store -- --ignored

Multi-tenancy: API keys → tenants, with full isolation

Single-key auth (with_api_key) covers the one-organization deployment. For a shared deployment serving several organizations, map API keys to tenants:

let config = ServerConfig::new("0.0.0.0:8080".parse()?, "./data/checkpoints")
    .with_tenant_key("acme", "sk-acme-…")     // (tenant, key) pairs
    .with_tenant_key("globex", "sk-globex-…")
    .with_api_key("sk-ops-…");                // optional: legacy key = `default` tenant

Key → tenant model. Every configured key maps to exactly one tenant. A request's X-Api-Key header is resolved to a tenant by the auth middleware (401 for missing/unknown keys) and everything the request touches is scoped to that tenant. With no keys configured the server is in open (dev) mode — no header required, all requests run as the default tenant, and behavior is byte-identical to pre-multi-tenancy versions.

Isolation semantics. Every tenant-scoped resource is namespaced per tenant: threads (including their checkpoint history and run bookkeeping), assistants, crons, and KV namespaces. Concretely, internal ids and namespaces carry a {tenant}/ prefix at the handler layer, so the storage backends separate naturally with no schema changes:

Resource JSON-file layout Postgres
Threads + checkpoints records: {store_path}/threads/{tenant}/{id}.json; checkpoints: {store_path}/{tenant}/{thread_id}/… server_threads.thread_id resp. rusty_checkpoints.thread_id = "{tenant}/{id}"
Assistants {store_path}/assistants/{tenant}/{id}.json server_assistants.assistant_id = "{tenant}/{id}"
Crons {store_path}/crons/{tenant}/{id}.json server_crons.cron_id = "{tenant}/{id}"
KV store {store_path}/store/{tenant}/{ns}/{key}.json server_kv.namespace = "{tenant}/{ns}"

The default tenant (open mode and the legacy with_api_key) is unprefixed, so existing deployments keep their flat layout and data — upgrading is a non-event. Tenant ids must match [A-Za-z0-9._-] (1–64 chars).

What this buys you:

  • Same external ids can coexist across tenants. acme and globex can both have a thread t1, an assistant bot, and a KV namespace memories without collisions — each resolves inside its own namespace.
  • Cross-tenant access is 404, never 403. Fetching another tenant's thread, run, assistant, cron, or KV item returns not_found, exactly as if the resource didn't exist. A 403 would confirm the resource exists (and that the id is worth attacking); a 404 leaks nothing.
  • The wire never shows internal prefixes. Responses always carry the external ids the client supplied; the {tenant}/ prefix is an internal storage detail.
  • The cron scheduler is tenancy-aware. It lists crons across all tenants but fires each one inside its owning tenant's namespace — cron-spawned threads and runs land in the right tenant.
  • GET /info stays tenant-neutral: it exposes only the service metadata and registered graphs — no tenants, keys, or resource counts.

Configuration

Configuration is code, via ServerConfig (constructed with ServerConfig::new(bind_addr, store_path) plus builder methods, or ServerConfig::default()). If you want twelve-factor env-based config in your binary, read the environment in your own main.rs and build the ServerConfig from it — the crate deliberately does not read process env itself.

Field / builder Default Purpose
bind_addr 0.0.0.0:8080 Listen address (used by serve)
store_path ./data/checkpoints JsonFileCheckpointer root ({store_path}/{thread_id}/{checkpoint_id}.json) and JSON-file platform persistence
with_postgres(…) None = JSON files Postgres URL for both checkpoints and the platform store (feature postgres; see Postgres persistence)
with_api_key(…) None = dev mode, no auth Static key required via the X-Api-Key header; maps to the default tenant
with_tenant_key(…) — Map an API key to a tenant for multi-tenant deployments (see Multi-tenancy)
with_max_concurrent_runs_per_thread(…) 1 Per-thread enqueue queue depth cap (there is always at most one active run per thread)
with_event_log_capacity(…) 1000 Per-run SSE replay buffer (frames)

Built-in limits (constants, not config knobs):

Limit Value Behavior
Terminal-run retention 1024 runs Run status/stream history is in-memory by design; the oldest terminal runs are evicted beyond the cap (active and queued runs are never evicted), after which GET /runs/{id} and GET /runs/{id}/stream answer 404
Blocking-wait ceiling 3600 s POST /threads/{id}/runs/wait answers 504 Gateway Timeout when the run hasn't terminated within the ceiling; the run itself keeps executing — poll GET /runs/{id}
Cron interval ceiling 31 536 000 s (1 year) interval_secs above the ceiling answers 400 (unbounded values would overflow the scheduler's timestamp math)

Deployment

Build one static binary:

cargo build --release
# -> target/release/my-agent   (statically linked; the bundled server_demo
#    measures ~5 MB as a macOS release build — feature sets shift the size)

Ship it in a scratch image — no interpreter, no pip layer, no system Python:

FROM rust:1-bookworm AS build
WORKDIR /app
COPY . .
RUN cargo build --release

FROM scratch                       # or gcr.io/distroless/static
COPY --from=build /app/target/release/my-agent /my-agent
ENTRYPOINT ["/my-agent"]

curl quickstart

With the server running locally in dev mode (no API key configured):

# Liveness + what's registered
curl localhost:8080/ok
# {"ok":true}
curl localhost:8080/info
# {"service":"rusty-server","version":"0.5.0","checkpointer":"json_file",
#  "store_path":"./data/checkpoints",
#  "graphs":[{"name":"react_agent","channels":["messages"]}]}

# Create a thread bound to a registered graph
curl -X POST localhost:8080/threads \
  -H 'Content-Type: application/json' \
  -d '{"graph": "react_agent"}'
# -> 201 {"thread_id": "3f2b9c…", "graph": "react_agent",
#         "metadata": null, "created_at": "2026-08-05T…Z"}

# Blocking run
curl -X POST localhost:8080/threads/$TID/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"input": {"messages": [{"role": "user", "content": "What is 17 + 25?"}]}}'

# Streaming run (SSE) — note -N to disable curl buffering
curl -N -X POST localhost:8080/threads/$TID/runs/stream \
  -H 'Content-Type: application/json' \
  -d '{"input": {"messages": [{"role": "user", "content": "Echo hi"}]},
       "stream_mode": ["updates", "values"]}'

# Resume an interrupted run (human-in-the-loop)
curl -X POST localhost:8080/threads/$TID/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"command": {"resume": {"approved": true}}}'

# Thread state + checkpoint history
curl localhost:8080/threads/$TID/state
curl -X POST localhost:8080/threads/$TID/history \
  -H 'Content-Type: application/json' -d '{"limit": 10}'

# Roll back a finished run's checkpoints
curl -X DELETE localhost:8080/threads/$TID/runs/$RUN_ID

# Time travel: fork the thread at an earlier checkpoint, replay the fork
CP_ID=$(curl -X POST localhost:8080/threads/$TID/history \
  -H 'Content-Type: application/json' -d '{"limit": 10}' \
  | jq -r '.[-1].checkpoint.checkpoint_id')
curl -X POST localhost:8080/threads/$TID/fork \
  -H 'Content-Type: application/json' \
  -d '{"checkpoint_id": "'$CP_ID'"}'
# -> 201 {"thread_id": "9c1e…", "checkpoints_copied": 1}
curl -X POST localhost:8080/threads/9c1e…/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"checkpoint": {"checkpoint_id": "'$CP_ID'"}}'

# Poll a background run's status (terminal runs carry output/error)
curl localhost:8080/runs/$RUN_ID

# Flight Recorder: the run's journaled evidence (RunEvents, seq order)
curl localhost:8080/runs/$RUN_ID/events
# …or download the run as a portable replay fixture for CI
curl localhost:8080/runs/$RUN_ID/fixture
# …or re-drive it server-side and verify the replayed evidence
curl -X POST localhost:8080/runs/replay \
  -H 'Content-Type: application/json' -d '{"run_id": "'$RUN_ID'"}'
# …or diff two runs' journals (e.g. a run vs its fork)
curl "localhost:8080/runs/diff?base=$RUN_ID&branch=$FORK_RUN_ID"

# Attach to a background run's SSE stream; reconnect with Last-Event-ID
# to skip frames you have already seen
curl -N localhost:8080/runs/$RUN_ID/stream
curl -N localhost:8080/runs/$RUN_ID/stream -H "Last-Event-ID: $LAST_FRAME_ID"

# Create an assistant and run by assistant_id
curl -X POST localhost:8080/assistants \
  -H 'Content-Type: application/json' \
  -d '{"name": "support-bot", "graph": "react_agent",
       "config": {"recursion_limit": 25}}'
curl -X POST localhost:8080/threads/$TID/runs/wait \
  -H 'Content-Type: application/json' \
  -d '{"assistant_id": "'$AID'"}'

# A cron that fires a run every 60 seconds on a fresh thread
curl -X POST localhost:8080/crons \
  -H 'Content-Type: application/json' \
  -d '{"graph": "react_agent", "interval_secs": 60,
       "input": {"messages": [{"role": "user", "content": "hourly summary"}]}}'
curl localhost:8080/crons
curl -X DELETE localhost:8080/crons/$CRON_ID

# Cross-thread KV store
curl -X PUT localhost:8080/store/memories/user-1 \
  -H 'Content-Type: application/json' -d '{"preference": "dark-mode"}'
curl localhost:8080/store/memories
curl -X DELETE localhost:8080/store/memories/user-1

With auth configured, add -H "X-Api-Key: $KEY" to every call. For a full walkthrough — project scaffolding, a two-node graph, streaming, and a complete interrupt/resume round trip — see docs/server-quickstart.md.

Roadmap

  • Phase A — the server crate (v0.1). GraphRegistry, the thread/run/SSE endpoint set, per-thread run queue with multitask_strategy (enqueue / reject) plus explicit rollback via DELETE /threads/{id}/runs/{run_id}, SSE with mode filters (updates / values / messages) + per-run event log + Last-Event-ID dedup, static API-key middleware, JsonFileCheckpointer wiring from ServerConfig::store_path. Implemented.
  • Phase C (partial) — platform surface (v0.2). GET /runs/{run_id} status polling, assistants (named graph + config aliases, JSON-persisted, assistant_id on run-create), crons (interval or 5-field cron schedules, durable records, background tokio scheduler firing runs on fresh threads, on_run_completed: keep|delete), and the cross-thread KV store (/store/{namespace}/{key}, JSON-file-backed). Implemented.
  • Phase C (continued) — time travel + Postgres persistence (v0.3). POST /threads/{id}/fork (full- or mid-history forks via core's Checkpointer::fork_thread), checkpoint replay on all run endpoints ("checkpoint": {"checkpoint_id": …} → RunConfig::with_checkpoint_id), and the postgres feature: ServerConfig::with_postgres(url) switches the run checkpointer to PostgresCheckpointer and the assistants/crons/KV surface to auto-migrated server_assistants / server_crons / server_kv tables behind a ServerStore trait. Implemented.
  • Phase C (continued) — permissive CORS (v0.3). router() layers tower_http::cors::CorsLayer::permissive(), so browser clients (the Studio) call the API cross-origin; preflights are answered before auth. Restrict for production — see CORS. Implemented.
  • Phase C (continued) — multi-tenancy (v0.4). API keys map to tenants (with_tenant_key(tenant, key); legacy with_api_key = the default tenant); threads + checkpoints, runs, assistants, crons, and KV namespaces are fully isolated via internal {tenant}/ id prefixing, with cross-tenant access answering 404 (never 403). Open mode and the default tenant keep the legacy flat storage layout. See Multi-tenancy. Implemented.
  • Phase C (continued) — hardening (v0.4). Durable thread records (threads/ JSON files / server_threads table — pre-restart checkpoints stay reachable through the API), the SSE attach endpoint (GET /runs/{id}/stream with Last-Event-ID replay), rollback guards (409 on the Postgres backend, on busy threads, and on mid-history suffix violations), the cron interval_secs clamp (≤ 1 year) + one-shot tombstones, reserved layout names rejected as client-chosen ids, the 1024-run retention cap, the 3600 s blocking-wait ceiling (504), and 400 for unknown history before cursors. Implemented.
  • R0.5 — Flight Recorder read surface (v0.5). Every run is journaled by the executor (core R0.5 kernel); the server persists the journal snapshot at every checkpoint boundary and at run completion ({store_path}/journals/{run_id}.json, or the auto-migrated server_journals table on Postgres) and serves it via GET /runs/{run_id}/events → {run_id, events, complete} in the golden-pinned RunEvent wire shape, with head-hash re-verification on read and 404/tenant-isolation semantics identical to GET /runs/{id}. GET /runs/{run_id}/fixture downloads the run as a portable ReplayFixture for CI replay. SDK parity: run_events(run_id) (Python) / runEvents(runId) (TypeScript). Implemented — the replay-POST endpoint lands in a later R0.5 wave.
  • Phase B — gRPC worker protocol (rusty-proto). RemoteNode: a gRPC client behind the same Node trait, delegating node execution to stateless out-of-process workers that long-poll named node-queues. The server keeps checkpoints, super-step scheduling, interrupts, and stream fan-out. Agent nodes are dominated by LLM latency (hundreds of ms to minutes), so a 1–5 ms gRPC hop is <1% overhead — and since State is already a JSON map, the wire boundary is lossless. Crash isolation, polyglot workers (a Python worker can host the LangChain ecosystem while Rust owns orchestration), and independent scaling of tool-heavy nodes follow.
  • Phase C (remainder). Thread listing/deletion endpoints, /metrics, and /graphs. (WasmNode is implemented in core rusty-agent-runtime v0.4 behind the wasm feature — register a Wasm-backed graph and this crate serves it unchanged.)

Deliberately skipped: A2A/MCP server endpoints and WebSocket "protocol v2" (SSE + HTTP sidecar is sufficient), and feedback_keys (LangSmith-tracing coupling we don't have).

License

Dual-licensed under either of

at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work by you, as defined in the Apache-2.0 license, shall be dual-licensed as above, without any additional terms or conditions.