# Salvor control-plane API
This is the HTTP and server-sent-events contract the Salvor control plane
serves. It is the surface the SDKs and the dashboard build against, so it
is specified here rather than left implicit in the handlers.
The server is a thin network layer over the durable runtime. It owns one event
store and constructs a runtime per request; every guarantee the CLI has (exact
replay, crash-safe resume, the write-ahead reconciliation rule) holds over HTTP
unchanged, because the same runtime enforces it. Clients stay thin: they submit
data and read events, and hold none of the durability themselves.
All request and response bodies are JSON, except the agent-definition body on
registration (which may be TOML) and the event stream (which is
`text/event-stream`). Every path is versioned under `/v1`.
## Auth
One optional shared-secret bearer token, the single-tenant posture. Two modes:
- **Token set.** Every request must carry `Authorization: Bearer <token>`. A
missing or wrong token is answered `401` with the standard error envelope.
- **No token.** The server trusts its caller; a reverse proxy is expected to
own auth. This is the default.
There is no user model and no role system. The `salvor serve --auth-token
<ENV_VAR>` flag names an environment variable holding the token, never the
token itself.
## Error envelope
Every error, whatever its status, has one shape:
```json
{ "error": { "code": "unknown_run", "message": "no run ... in this store" } }
```
`code` is a stable machine token; `message` is a human sentence. A
`details` object is present only when there is structured evidence: the
reconciliation refusal (`details.intent`, below) and an invalid graph submission
(`details.errors`, the full node/edge-precise list). The status codes and their
codes:
| 400 | `bad_request` | Malformed body, bad run id, or a resume input the recorded schema rejects |
| 401 | `unauthorized` | Missing or wrong bearer token |
| 404 | `unknown_run` | No run under that id |
| 404 | `unknown_agent` | No agent registered under that id (also: a graph run references an unregistered agent) |
| 400 | `invalid_graph` | A submitted graph document failed strict validation; `details.errors` carries the complete node/edge-precise list |
| 400 | `approval_schema_violation` | A graph run parked at a `gate` was resumed with an approval its `approval_schema` does not describe; `details.node` names the gate and `details.violations` lists every violation. Nothing is recorded; the run stays parked |
| 404 | `unknown_graph` | No graph stored under that hash |
| 409 | `not_a_graph_run` | The per-run graph projection (or a fork) was asked for an ordinary agent run |
| 409 | `invalid_fork_node` | A fork named a node the origin never entered |
| 409 | `origin_needs_reconciliation` | A fork of an origin parked at a dangling write; `details.intent` carries the origin's recorded write |
| 409 | `write_replay_hazard` | A fork would re-execute recorded writes the operator has not acknowledged; `details.writes` lists exactly the ones still needing acknowledgement |
| 409 | `run_exists` | Starting a run at an id that already has history |
| 409 | `wrong_state` | A verb applied to a run in the wrong state (resolving a run with no dangling write) |
| 409 | `client_driven_run` | Resuming a run opened through `/v1/client-runs`; its client is the only legal driver, so its own re-open is what resumes it |
| 409 | `needs_reconciliation` | Resuming a run whose log ends at a write intent with no completion; `details.intent` carries the recorded write |
| 409 | `still_sleeping` | Resuming a run parked on a durable timer before its instant; `details.wake_at` and `details.remaining_seconds` say when it can be driven. Nothing is recorded |
| 401 | `missing_drive_token` | A client-driven append with no drive token (see [Client-driven runs](#client-driven-runs)) |
| 403 | `invalid_drive_token` | A client-driven append whose drive token is not the run's current lease, or a release asking to end a hold the caller does not hold |
| 409 | `lease_held` | Re-opening a client-driven run whose driver still holds a current lease; `details.lapses_in_seconds` says how long until the hold lapses. Nothing is recorded and no lease is minted |
| 409 | `divergence` | A client-driven append that is not the legal next event, or different bytes at an already-recorded position |
| 422 | `unsupported_event_kind` | A client-driven append carrying a model or tool event, which this surface does not accept |
| 413 | `payload_too_large` | A client-driven append over the body-size or per-batch cap |
| 503 | `model_executor_unavailable` | A model step against a server with no model executor wired; no intent is written |
| 502 | `model_execution` | A model step's provider call failed; no completion is recorded, so the intent is left dangling |
| 404 | `unknown_tool` | A tool step naming a tool the server's registry does not hold; no intent is written |
| 503 | `tool_registry_unavailable` | A tool step against a server with no tool registry wired; no intent is written |
| 502 | `tool_execution` | A tool step's dispatch failed; no completion is recorded, so the intent is left dangling |
| 500 | `internal` | A store read or agent build failed unexpectedly |
## Endpoints
| POST | `/v1/agents` | Register and validate an agent definition |
| GET | `/v1/agents` | List registered agent ids |
| GET | `/v1/agents/{hash}` | Read one registered definition back |
| POST | `/v1/runs` | Start a run |
| GET | `/v1/runs` | List runs with folded status |
| GET | `/v1/runs/{id}` | Get one run's derived state |
| GET | `/v1/runs/{id}/replay` | Dry-run replay: the derived state, executing nothing |
| GET | `/v1/runs/{id}/events` | Stream a run's events (server-sent events) |
| POST | `/v1/runs/{id}/resume` | Continue a run (resume a parked one, recover a crashed one); works on a graph run unchanged |
| POST | `/v1/runs/{id}/resolve` | Record a dangling write's completion by hand |
| POST | `/v1/runs/{id}/abandon` | Retire a run by hand, appending a terminal `RunAbandoned` |
| GET | `/v1/runs/{id}/graph` | A graph run's per-node projection (for the canvas) |
| POST | `/v1/runs/{id}/fork` | Fork a graph run from a node boundary into a new run (refuse-then-record) |
| GET | `/v1/runs/{id}/forks` | The forks of a run, as a derived index |
| GET | `/v1/capabilities` | What this build of the control plane can do (a dashboard probes it) |
| POST | `/v1/graphs` | Submit and strictly validate a graph document |
| GET | `/v1/graphs` | List stored graphs with a shape summary |
| GET | `/v1/graphs/{hash}` | Read one stored graph document back |
| POST | `/v1/graphs/validate` | Validate a document without storing it |
| POST | `/v1/graph-runs` | Start a run of a stored graph |
| GET | `/v1/client-tools` | List the client-performed tool declarations this server holds |
| POST | `/v1/client-runs` | Open or resume a client-driven run |
| POST | `/v1/client-runs/{id}/release` | Hand the drive-token lease back, so the next open takes the run at once |
| POST | `/v1/client-runs/{id}/heartbeat` | Refresh the lease without driving, for a driver inside a long body |
| GET | `/v1/client-runs/{id}/log` | Read a client-driven run's recorded log |
| POST | `/v1/client-runs/{id}/events` | Append control and context events (the guarded append) |
| POST | `/v1/client-runs/{id}/model-step` | Perform and record a model call (server-performed) |
| POST | `/v1/client-runs/{id}/tool-step` | Perform and record a tool call (server-performed) |
| POST | `/v1/client-runs/{id}/client-tool-intent` | Open a tool call the CLIENT performs, and derive its idempotency key |
| POST | `/v1/client-runs/{id}/client-tool-completion` | Record what a client-performed tool call returned |
| POST | `/v1/client-runs/{id}/client-model-intent` | Open a model call the CLIENT performs, or replay a recorded one |
| POST | `/v1/client-runs/{id}/client-model-completion` | Record what a client-performed model call returned |
| POST | `/v1/client-runs/{id}/resolve` | Record a dangling write's completion by hand (client-driven) |
### POST /v1/agents
Register a definition. Under the single built-in loop an agent is pure data, so
it has a content hash (`agent_def_hash`, the same id every `RunStarted` event
records). A definition is submitted once and referenced by that hash from then
on, so a start payload carries only a hash and an input, and the same
definition drives every start, resume, and recover.
Registration also validates: the server builds the agent (which spawns and
immediately closes any MCP sessions) to confirm it is buildable and to compute
the hash. A definition that will not build is a `400`, which also covers an
out-of-bounds `name` (see below): the server re-validates it at this same
build step, the same as any other client-supplied config, rather than trusting
whatever a submitter's own tooling already checked.
- Request: the definition body. `Content-Type: application/toml` for the agent
TOML the CLI reads, or `application/json` for the same fields as JSON. Both
accept an optional top-level `name`: a short display label (at most 64
characters, and not empty or all whitespace when set) shown by tooling that
resolves `agent_def_hash` back to something readable. `name` plays no part
in `agent_def_hash`, so renaming an agent (registering the same definition
again under a different `name`) never mints a new identity.
- Response `201`:
```json
{ "agent": "sha256:34e0...", "created": true }
```
`created` is `false` when the identical definition was already registered.
The registry is process-local. After a restart, re-register definitions; the
hash is stable, so runs that recorded a reference to it still resolve.
### GET /v1/agents
```json
{ "agents": [ { "agent": "sha256:34e0..." }, { "agent": "sha256:9f2c...", "name": "support-triage" } ] }
```
### GET /v1/agents/{hash}
```json
{ "agent": "sha256:34e0...", "format": "toml", "definition": "model = ..." }
```
or, when the definition declared a `name`:
```json
{ "agent": "sha256:9f2c...", "format": "toml", "definition": "model = ...\nname = \"support-triage\"\n",
"name": "support-triage" }
```
`name` is present only when the registered definition actually declared one;
there is no meaningful empty name to fall back to, so an agent registered with
none omits the field entirely from both this response and the list above,
rather than emitting `"name": null`. This is the same honest-absence rule
[`GET /v1/runs`](#get-v1runs) already applies to `agent_def_hash` and
`labels`.
`404 unknown_agent` when the hash is not registered.
### POST /v1/runs
Start a fresh run. Returns at once with the run id; the run then drives in the
background (see [Driving a run](#driving-a-run)).
- Request:
```json
{ "agent": "sha256:34e0...", "input": <any json>, "run_id": "<uuid, optional>",
"labels": { "build": "42", "env": "prod" } }
```
`input` defaults to `null`. `run_id` is optional; when omitted the server mints
one. Passing one lets a client choose the id (a UUID).
`labels` is optional: free-form correlation tags recorded once on the run's
`RunStarted` event, so runs can be told apart in a list (which resume build
was this). At most 16 labels, each key at most 64 bytes and each value at
most 256 bytes; a label is a tag, not a payload. A run started with no
`labels` records none, and that is a different, honest thing from an empty
object (see [`GET /v1/runs`](#get-v1runs) below): omit the field rather than
send `{}` unless an explicit empty set is genuinely what is meant.
- Response `201`:
```json
{ "run": "6f...uuid", "status": "running" }
```
- `400 bad_request` when `labels` violates the bounds above. Checked before the
run is spawned, so a rejected request creates no run at all.
- `404 unknown_agent` when the agent is not registered.
- `409 run_exists` when the chosen `run_id` already has history.
### GET /v1/runs
```json
{ "runs": [
{ "run": "6f...", "status": { "state": "running" },
"event_count": 10, "first_recorded_at": "2026-...", "last_recorded_at": "2026-...",
"usage": { "input_tokens": 250, "output_tokens": 50 },
"step_count": 2,
"agent_def_hash": "sha256:34e0...",
"labels": { "build": "42", "env": "prod" },
"driver": "attached" }
] }
```
Status is folded from each log, not stored, so it is always current.
`usage`, `step_count`, `agent_def_hash`, and `labels` are additive fields,
folded from the same per-run log read and fold `status` has always come from,
so listing does not read a run's log twice. `usage` is the same shape as
[`GET /v1/runs/{id}`](#get-v1runsid)'s `usage`. `step_count` is how many
`ModelCallRequested` events the run's log holds. `agent_def_hash` is the hash
recorded on the run's `RunStarted` event, the same value
[`POST /v1/agents`](#post-v1agents) returned when the definition was
registered. `labels` is whatever correlation tags were recorded on
`RunStarted` at [`POST /v1/runs`](#post-v1runs) time.
**Honest absence, not zero.** `usage`, `step_count`, and `agent_def_hash` are
present, and are real counts, whenever a run's log folds: a run with no model
calls yet reports a true `step_count: 0` and `usage` of all zeros, not a
missing field, because that zero is known. They are *absent* (omitted from the
object entirely, per `skip_serializing_if`, never `null` and never `0`) only
when a run's log cannot be read at all (a corrupt or unreadable stored
envelope). That failure is scoped to the one run whose row it is: the store's
per-run summary (`event_count`, `first_recorded_at`, `last_recorded_at`) is a
cheap aggregate that never touches the row's JSON payload, so it, and even
`status`, which also depends on the unreadable log, are the only fields such
a run's entry carries. Before this fold ran, a single unreadable log failed
the whole listing (`500`); now it degrades only that one entry, so this is
additive: old consumers reading only the pre-existing fields see the exact
same JSON for every run whose log reads cleanly.
`labels` follows the same absence rule, one step further: it is omitted both
when a run recorded no labels at all (an unlabeled run, or one from before
labels existed) *and* when it recorded an explicit empty set. The API never
emits `"labels": {}`, because an empty map is not a fact worth asserting any
more than an unknown count is. A run that recorded at least one label reports
exactly what was recorded.
**Liveness evidence: `driver`.** `driver` reports whether a driver is currently
running the run: `"attached"` or `"none"`. It reads no log. It consults only
the process's own driving-run set and client-run leases, the truth the server
already holds:
- A **server-driven** run is `"attached"` exactly when a driver task is still
running it in this process, the same fact the event stream's `detached`
end-frame reports (see [`GET /v1/runs/{id}/events`](#get-v1runsidevents)). The
task is dropped the instant it ends (completes, parks, or errors), so this is
exact, not a heuristic.
- A **client-driven** run is `"attached"` exactly when this process holds a
current lease for it: the driver presented its drive token within the lease
TTL (default 60s; `salvor serve` reads `SALVOR_CLIENT_LEASE_TTL_SECS`). A
lapsed lease (the tab closed, the SDK exited) is `"none"`. It is the same
lease, read the same way, that decides whether re-opening the run is refused
with `409 lease_held` (see [The drive token](#the-drive-token)), so a run that
reports `"attached"` here is exactly a run another driver cannot take.
`driver` follows the same zero-vs-absent rule as the folded fields, one way: it
is **omitted entirely for a terminal run** (`completed` or `failed`), because
whether a driver is attached to a finished run is not a meaningful question, and
`"driver": "none"` would assert a placeholder where there is no fact. Every
non-terminal run reports a real `"attached"` or `"none"`; a run whose log could
not be read omits it along with `status` (there is no folded status to gate it).
`driver` is **evidence, not a verdict.** Paired with `last_recorded_at` (the
newest envelope's timestamp, the "when did anything last happen" fact), it is
what a dashboard reads to derive a *stalled* state: a run that folds to
`running` yet reports `"driver": "none"` and whose `last_recorded_at` has gone
stale is going nowhere: resolved but never re-driven, its driver crashed, or
its client abandoned. The server reports the two facts; the client derives the
stalled verdict, the same division of labor `status` itself has.
### GET /v1/runs/{id}
The run's derived state:
```json
{
"run": "6f...",
"status": { "state": "suspended", "reason": "...", "input_schema": { ... } },
"event_count": 6,
"usage": { "input_tokens": 250, "output_tokens": 50 },
"pending": { "kind": "tool", "seq": 5, "tool": "record", "input": ...,
"effect": "write", "idempotency_key": null },
"first_recorded_at": "2026-...",
"last_recorded_at": "2026-...",
"driver": "attached"
}
```
`404 unknown_run` when the id has no history and no run is being driven under it.
`driver` is the same liveness evidence [`GET /v1/runs`](#get-v1runs) carries,
under the same rules: `"attached"` / `"none"` for a non-terminal run, omitted
for a terminal one. A run with no history yet but a driver task already spawned
reports `"driver": "attached"` alongside its `running` status.
#### The status object
Always `{ "state": "<name>", ... }`:
| `not_started`, `running`, `awaiting_model`, `awaiting_tool`, `needs_reconciliation` | none |
| `suspended` | `reason`, `input_schema`, `kind` (only `"signal"`, and only when present) |
| `sleeping` | `wake_at` (RFC 3339); once the server's clock is past it, also `overdue` (`true`) and `overdue_seconds` (whole seconds since `wake_at`) |
| `budget_exceeded` | `budget` (`{kind, limit}`), `observed` |
| `completed` | `output` |
| `failed` | `error` |
| `abandoned` | `reason` (when given), `unresolved_write` (`{seq, tool}`, only when a needs-reconciliation run was abandoned) |
On `suspended`, `kind` says what the run is waiting on when it is not a person:
`"signal"` means an external system (a webhook, a callback) will resume it, so
it is nobody's task and belongs nowhere near an approval inbox. The key is
absent for a human gate, which is what every suspension without it means.
It resumes exactly the way anything resumes a run: whatever holds the bearer
token calls `POST /v1/runs/{id}/resume` (the CLI equivalent is `salvor resume`)
with a payload the recorded `input_schema` accepts. There is no separate
webhook endpoint and no per-run secret, so nothing stops a resume that arrives
before the real signal; the token is the whole boundary (see the README's
"Operating it" section and `SECURITY.md`'s single shared secret). A signal
wait carries no deadline of its own and waits until something resumes it; an
operator can stand in for a signal that never comes with `salvor resume` or
the Bridge Inspector's "Awaiting a signal" band, the only place the Bridge
offers that action, since a signal wait never appears in the Inbox.
#### The pending object
`null`, or one of:
```json
{ "kind": "model", "seq": 3, "request_hash": "sha256:..." }
{ "kind": "tool", "seq": 5, "tool": "...", "input": <json>,
### GET /v1/runs/{id}/replay
The dry-run replay projection: the full derived state as a pure fold of the
recorded log, executing nothing. This is what the CLI's `replay --dry-run`
prints, as JSON.
```json
{
"status": { "state": "completed", "output": ... },
"usage": { "input_tokens": 250, "output_tokens": 50 },
"next_seq": 10,
"pending": null
}
```
`404 unknown_run` when the id has no history.
### GET /v1/runs/{id}/events
The event stream. `Content-Type: text/event-stream`.
#### Framing
Every recorded event is one frame:
```text
id: 4
data: {"run_id":"6f...","seq":4,"schema_version":1,"recorded_at":"...","event":{"kind":"ToolCallCompleted","payload":{...}}}
```
- `data` is exactly the pinned event-envelope wire JSON, the same bytes
`GET /v1/runs/{id}` events come from and `salvor history --json` prints, so a
client decodes stream frames and log rows with one parser.
- `id` is the event's sequence number.
- Envelope frames carry no `event:` field, so a browser `EventSource` receives
them through `onmessage`.
- A `Suspended` event's payload carries the same optional `kind` the status
object does: `"signal"` with the same meaning and absence rule described
under [the status object](#the-status-object), so an SSE consumer can build
the same filter the Bridge does. The full payload is `{"reason": "...", "input_schema": {...}}` for a human gate and `{"reason": "...", "input_schema": {...}, "kind": "signal"}` for a signal wait.
- When the run reaches a resting point (completed, failed, abandoned,
suspended, sleeping, budget-exceeded, or needs-reconciliation) the stream
sends one final frame with `event: end` carrying the status it rested at,
then closes:
```text
event: end
data: {"status":{"state":"completed","output":...}}
```
A run parked on a durable timer rests too, and its frame carries the deadline:
```text
event: end
data: {"status":{"state":"sleeping","wake_at":"2026-08-14T09:00:00Z"}}
```
Nothing drives a sleeping run, and its nap is measured in hours or days, so
the stream closes rather than polling for the duration. `wake_at` is when to
open a fresh stream; the events the wake records are read by that one.
If the driving task was killed and no driver is running the run in this process,
the end frame also carries `"detached": true`; recovering the run opens a fresh
stream that tails the continuation.
#### Replay then live tail
On connect the server sends every recorded event at or after the cursor, then
polls the store and sends new events as they land, until the resting frame. A
log is append-only with contiguous ascending sequence numbers, so the stream is
gap-free and duplicate-free by construction.
#### Cursor: resuming a dropped stream
A dropped connection resumes without gaps or duplicates:
- **`Last-Event-ID` header.** A browser `EventSource` resends the last `id` it
saw. The server resumes from that sequence plus one.
- **`?from_seq=<n>` query.** A non-browser client that tracks its own position
asks for events from sequence `n` onward.
`Last-Event-ID` wins when both are present. With neither, the stream starts at
sequence 0 (a full replay).
`404 unknown_run` when the id has no history and no run is being driven under it.
### POST /v1/runs/{id}/resume
Continue a run. The server reads the run's derived state and dispatches on it,
the same mapping `salvor resume` uses:
- **Client-driven** (opened through `/v1/client-runs`): refused `409
client_driven_run` before anything else, whatever the run's state folds to.
That run's client holds the single-writer drive token and is its only legal
driver; resuming it here would start a second writer racing the client's
lease, even when the agent it recorded happens to be registered on this
server too. Its client resumes it by re-opening `POST /v1/client-runs`.
- **Parked** (suspended or budget-exceeded): the request must carry an `input`,
validated against the recorded suspension schema or the budget-extension
shape before anything is recorded. The run then resumes in the background.
- **Crashed** (running, or interrupted mid model or tool step): the run
recovers with no input. An `input` in the body is ignored.
- **Sleeping**: past its `wake_at`, it re-drives exactly as a crashed run does,
which is what waking is. Before its `wake_at`, refused `409 still_sleeping`
with the deadline as evidence (see below).
- **Needs reconciliation**: refused `409`, with the recorded write intent as
evidence (see below). Use `resolve` to move past it.
- **Finished** (completed or failed): reported, `200`, left alone.
- Request (optional body):
```json
{ "input": <any json> }
```
- Response `202` for a run now driving:
```json
{ "run": "6f...", "status": "running", "outcome": "driving" }
```
- Response `200` for an already-finished run:
```json
{ "run": "6f...", "outcome": "completed", "status": { "state": "completed", "output": ... } }
```
- `400 bad_request` when a parked run is resumed with no input, or with an
input the recorded schema rejects. `{"input": null}` is an input of `null`,
not an absent one: it reaches validation like any other value.
- `400 approval_schema_violation` when a graph run parked at a `gate` is resumed
with an approval the gate's `approval_schema` does not describe. The check
runs before anything is spawned or appended, so the run is still parked at the
gate and a corrected approval resumes it:
```json
{ "error": {
"code": "approval_schema_violation",
"message": "the approval does not satisfy gate `approve`'s approval_schema; the run is still parked at that gate, so a conforming approval resumes it",
"details": {
"node": "approve",
"violations": [ { "path": "$.approved", "message": "\"yes\" is not of type \"boolean\"" } ]
}
} }
```
An `approval_schema` that names `required` or `properties` without a `type` is
read as asking for an object, so a bare `null`, number, or string is refused
even though plain JSON Schema semantics would let it pass vacuously.
- `409 needs_reconciliation`:
```json
{ "error": {
"code": "needs_reconciliation",
"message": "run ... needs reconciliation: a write was recorded but never completed ...",
"details": { "intent": {
"kind": "tool", "seq": 4, "tool": "charge", "input": { "amount": 10 },
"effect": "write", "idempotency_key": null, "recorded_at": "2026-..."
} }
} }
```
- `409 still_sleeping` when the run is parked on a durable timer whose instant
has not arrived. Nothing is recorded and no driver is spawned, so the run is
exactly as asleep as it was; retry at `wake_at` or later, or leave it to the
wake sweeper if this server holds the agent or graph the run recorded, or run
`salvor wake` with the run's files if it does not:
```json
{ "error": {
"code": "still_sleeping",
"message": "run ... is sleeping until 2026-08-14T09:00:00Z and cannot be resumed for another 1740s. It is not waiting on input: it continues when its deadline passes and something re-drives it: this server's wake sweeper, when it holds the agent or graph the run recorded, or salvor wake with the run's files",
"details": { "wake_at": "2026-08-14T09:00:00Z", "remaining_seconds": 1740 }
} }
```
- `404 unknown_agent` when the agent the run started under is not registered on
this server (re-register it, then resume).
- `409 client_driven_run` when the run was opened through `/v1/client-runs`:
```json
{ "error": {
"code": "client_driven_run",
"message": "run ... is client-driven; its client resumes it by re-opening POST /v1/client-runs, not this endpoint, so this server never becomes a second writer against the client's lease"
} }
```
To watch the continuation, open the event stream after a `202`.
### POST /v1/runs/{id}/resolve
Record the completion of a dangling write by hand, the operator side of
reconciliation. After a human has verified externally what a recorded-but-never
-completed write did, this records the completion they observed, so replay
treats the call as done and never re-runs it. It records exactly one event and
drives nothing.
- Request:
```json
{ "output": <the json the tool returned> }
```
- Response `200`:
```json
{ "run": "6f...", "resolved": true, "status": { "state": "running" } }
```
- `409 wrong_state` when the run does not need reconciliation (there is no
dangling write to resolve). On a client-driven run the message says what the
unfinished call actually needs: an unfinished read or model call is performed
again by the client on its next drive, and needs nobody. It never names
`recover`, which is a server-driven verb that `POST /v1/runs/{id}/resume`
refuses a client-driven run outright.
- `404 unknown_run` when the id has no history.
**The output is checked against the declaration, when the call was the
client's.** A dangling `ToolCallRequested` carrying `performed_by: "client"` is
a call this server never witnessed, and the operator's declaration is the only
thing that says what a finished one looks like. So the same rules the completion
boundary applies to a client's own report apply to a hand-recorded output
before it is written:
- no declaration loaded for the tool: `400 bad_request`, naming the tool and
`--client-tool`. A stale registry is the operator's to fix, and recording an
unexamined output is the one thing this path must not do;
- the output fails the declaration's `output_schema`: `400 bad_request`;
- a `require_equal` field's value differs from the one the intent recorded: `400
bad_request`, naming the tool and the field. A run whose log says a 5000 payout
was authorized must not end up carrying 50000 as its result, whoever typed it.
`trust_completion` is NOT checked here. It answers "may the client close this
call", and the point of a `false` is that a person closes it instead, which is
what this path is. A call salvor performed itself is not checked at all: this
server witnessed it and holds no declaration for it.
**The recorded completion names its settler.** It carries `settled_by:
"operator"`, the one thing a hand-recorded completion says that an ordinary one
does not: a person put this output here, over the run's head, and nothing in
this process witnessed the call it closes. Replay never reads the field and a
resolved completion replays as the output it records, exactly as it did before
the field existed; `salvor log` renders it as `[Operator]`, in the same
bracketed register a client-performed intent renders as `[Client]`. The app's
own completions omit the key entirely.
**It clears a client-driven run's lease.** A dangling write is a driver that
never came back to record what its write did, and this caller presents no drive
token, so it is not that driver. Recording the completion over its head
therefore says the driver is gone, and a recorded resolution drops the lease it
left behind: the run's own client re-opens it on the next request rather than
being refused `409 lease_held` for the rest of the TTL right after an operator
unstuck it. Only the lease goes; the run keeps its recorded `driven_by:
"client"`, so it is still a client-driven run to every surface that reads the
log. On a server-driven run there is no lease and nothing changes.
`salvor resolve` on the command line cannot do this. It writes the store
directly and has no way to reach a running server's memory, so a lease held by a
live server survives a CLI resolve and lapses on its own; the middleware's next
open then waits at most the lease TTL
(`SALVOR_CLIENT_LEASE_TTL_SECS`, 60s by default). Resolve over HTTP against a
live server to free the run at once.
### POST /v1/runs/{id}/abandon
Retire a run by hand. A deliberate sibling of `resolve`: the operator's "we do
not care about this run anymore" path, for a run that is dead forever or no
longer worth carrying in the inbox. It validates the run is non-terminal,
appends one terminal `RunAbandoned` event server-stamped through the append
guard, and returns the receipt: the appended seq and the re-derived status.
Abandonment is **not** failure: `RunFailed` is untouched, and the run derives to
its own terminal `abandoned` status, treated as terminal (never attention)
everywhere downstream.
**Why no lease.** Abandon is an operator action over the store, not a step in
driving the run, so it presents no drive token and needs no lease, unlike a
client-driven append. It works for any run in the store whatever drove it (a
server task, a client SDK, or nothing at all anymore); the very case it exists
for is a run no driver is coming back to. The append guard's terminal rule is
the only concurrency protection it needs: a run that reached a terminal first
refuses the abandonment.
- Request (optional body; an empty body abandons with no reason):
```json
{ "reason": "husk is dead forever" }
```
- Response `200`:
```json
{ "run": "6f...", "abandoned": true, "appended_seq": 7,
"status": { "state": "abandoned", "reason": "husk is dead forever" } }
```
- **Needs reconciliation is allowed, not refused.** Abandoning a run parked at a
dangling write is the case abandonment most needs to serve. The endpoint
computes the outstanding write from the log's dangling intent and records it on
the event, so the terminal status carries an `unresolved_write` and never
claims the write question was answered:
```json
{ "run": "6f...", "abandoned": true, "appended_seq": 5,
"status": { "state": "abandoned",
"unresolved_write": { "seq": 4, "tool": "charge" } } }
```
- `409 wrong_state` when the run is already terminal (completed, failed, or
previously abandoned); there is nothing left to abandon.
- `404 unknown_run` when the id has no history.
## Graphs and graph runs
A graph document is a control document: an acyclic set of nodes (`agent`,
`tool`, `gate`, `branch`, `map`, `fold`, `delay`) authored once, submitted,
hashed, and frozen for a run. Every node payload may carry an optional `name`: a short display
label (at most 64 characters, and, when set, not empty or all whitespace;
`400 invalid_graph` reports a violation node-precise as `node_name_too_long`
or `blank_node_name`). Unlike an agent definition's own `name` (excluded from
its `agent_def_hash` so a rename never mints a new agent identity), a node's
`name` is an ordinary field on the payload and hashes like any other: a graph
document IS its content hash, so renaming a node is authoring a new document
version, by design. A graph run is an ordinary run with a richer log: its head is
`GraphRunStarted` instead of `RunStarted`, and its nodes narrate the walk, so
[`GET /v1/runs/{id}`](#get-v1runsid), [`/replay`](#get-v1runsidreplay),
[`/events`](#get-v1runsidevents), the enriched [`GET /v1/runs`](#get-v1runs)
list, and [`POST /v1/runs/{id}/resume`](#post-v1runsidresume) all work on it
through their existing code. A graph run has no single `agent_def_hash` (it
coordinates many), so that field is simply absent from its run-list entry:
honest absence, exactly as `labels` is absent when a run recorded none.
### Resolution and the tool story
Starting a graph run resolves what the document references against the server's
live inventory, synchronously, before the run is spawned: every `agent` node's
hash must be a **registered agent** (built through the same factory an agent run
uses), and every `tool` node's tool must be present in the server's **tool
registry**, the SAME registry a client-driven [tool step](#post-v1client-runsidtool-step)
dispatches through. No separate tool-registration surface exists.
`salvor serve` wires that registry EMPTY, so on a stock server every `tool` node
is a precise `404 unknown_tool` until a host registers the tool it names.
### POST /v1/graphs
Submit a graph document (the body is the document JSON). The server validates it
strictly and all at once (collect-all, no short-circuit), and stores it
content-addressed by its reproducible hash.
- Response `201`:
```json
{ "graph": "sha256:...", "created": true }
```
`created` is `false` when the identical document was already stored
(re-submitting is idempotent: same document, same hash).
- `400 invalid_graph` on any validation failure. `details.errors` is the
complete list; each entry has a `code`, a `message`, and the node or edge it
names, for example:
```json
{ "error": { "code": "invalid_graph", "message": "the graph document has 1 validation error(s)",
"details": { "errors": [
{ "code": "dangling_edge", "message": "edge `approve` -> `ghost` references unknown node id `ghost`",
"edge": { "from": "approve", "to": "ghost" }, "missing": "ghost", "suggestion": null }
] } } }
```
A document that does not even parse strictly (an unknown field, a missing one)
is one `invalid_graph` error with code `malformed_document`.
### GET /v1/graphs
```json
{ "graphs": [ { "graph": "sha256:...", "node_count": 3, "edge_count": 2,
"entry_nodes": ["research"], "terminal_nodes": ["publish"] } ] }
```
### GET /v1/graphs/{hash}
```json
{ "graph": "sha256:...", "document": { "schema_version": 1, "nodes": [ ... ], "edges": [ ... ] } }
```
`404 unknown_graph` when nothing is stored under the hash.
### POST /v1/graphs/validate
Validate a document without storing it: submit's dry run, the graph counterpart
of `/replay`. It always answers the question rather than treating an invalid
document as a bad request:
- Response `200`, valid:
```json
{ "valid": true, "graph": "sha256:...", "summary": { "node_count": 3, "edge_count": 2,
"entry_nodes": ["research"], "terminal_nodes": ["publish"] } }
```
- Response `200`, invalid: `{ "valid": false, "errors": [ ... ] }`, the same
node/edge-precise list `POST /v1/graphs` refuses with. Nothing is ever stored.
### POST /v1/graph-runs
Start a run of a stored graph and return its id at once (the same fire-and-return
shape [`POST /v1/runs`](#post-v1runs) uses).
- Request:
```json
{ "graph_hash": "sha256:...", "input": { ... }, "labels": { "build": "42" } }
```
`input` defaults to `null`; `labels` is optional (same bounds as an agent run's).
- Response `201`: `{ "run": "6f...", "status": "running" }`.
- `404 unknown_graph` when the hash names no stored graph.
- `400 bad_request` when `labels` violates the bounds.
- `404 unknown_agent` (naming the node) when an `agent` node references an
unregistered agent; `404 unknown_tool` (naming the node) when a `tool` node
names a tool the registry does not hold. Both are resolved up front, so a run
is spawned only once everything it references resolves.
A parked graph run (a `gate`, a budget crossing) resumes through the ordinary
[`POST /v1/runs/{id}/resume`](#post-v1runsidresume): the server re-drives it over
the engine, looking the graph document back up by the hash the log records. A run
parked at a `gate` has its approval checked against that gate's `approval_schema`
first, and a non-conforming one is `400 approval_schema_violation` naming the node
and listing every violation, with nothing recorded and the run left parked.
### GET /v1/runs/{id}/graph
A graph run's per-node projection, for the canvas: which nodes the walk has
reached, which case each `branch` fired, and any `map` fan-out. Absent-vs-null
throughout: a node's `branch_case` and `map` appear only when recorded, and a
node the walk has not reached is simply absent (distinct from a `skipped` one).
```json
{ "graph_hash": "sha256:...", "current_node": "approve", "nodes": [
{ "node": "research", "state": "exited" },
{ "node": "approve", "state": "entered" },
{ "node": "reject", "state": "skipped", "reason": "no live inbound edge: an upstream branch routed to another case" }
] }
```
`state` is `entered`, `exited`, or `skipped` (with a `reason`). `current_node` is
present only while a node is entered and not yet exited. A forked run's
projection also carries a `forked_from` object (the `ForkOrigin` record:
`run_id`, `through_seq`, `from_node`, `graph_hash`, `acknowledged_writes`).
- `404 unknown_run` when the id has no history.
- `409 not_a_graph_run` when the run is an ordinary agent run (no
`GraphRunStarted` head), mirroring the other `409` shapes.
### POST /v1/runs/{id}/fork
Fork a graph run from a node boundary into a NEW run, and (the differentiator)
refuse to re-execute a recorded write the operator has not acknowledged.
A fork is a new run whose log opens with the origin's prefix (every event below
the fork node's `NodeEntered`) rewritten under the fork's own id, its seq-0
`GraphRunStarted` carrying `forked_from`. The origin is never touched. The child
then continues from the fork node exactly as a recovered graph run does. A fork
reuses the origin's graph unchanged (it may not edit it): to change the graph,
submit a new document and start a fresh run.
- Request:
```json
{ "from_node": "publish", "acknowledge_writes": [4], "dry_run": false }
```
`from_node` is the node boundary to restart from. `acknowledge_writes` (default
`[]`) are the origin log positions of the `Effect::Write` intents in the
re-walked segment the operator accepts may re-fire; they must cover the full
hazard set. `dry_run` (default `false`) previews without creating a run.
- Response `201`: `{ "run": "<child>", "status": "running", "forked_from": {
"run_id": "<origin>", "through_seq": 3, "from_node": "publish", "graph_hash":
"sha256:...", "acknowledged_writes": [4] } }`. The `acknowledge_writes` seqs
are recorded permanently into the child's `forked_from.acknowledged_writes`.
- `409 write_replay_hazard` when the re-walked segment holds unacknowledged
writes (the refuse-then-record refusal):
```json
{ "error": {
"code": "write_replay_hazard",
"message": "forking run ... would re-execute 1 recorded write(s) ...",
"details": { "writes": [
{ "seq": 4, "tool": "publish", "input": { ... },
"idempotency_key": null, "recorded_at": "2026-..." }
] }
} }
```
`details.writes` lists exactly the writes still needing acknowledgement (all of
them on a first, bare fork; a partial acknowledgement narrows it to what is
missing). Acknowledging every listed `seq` lets the fork proceed. Idempotent
tools are NOT listed: a graph tool's idempotency key is derived from its
position in the graph (graph hash, node id, call index), so a fork presents the
same key its origin recorded and the provider collapses the duplicate, so
`Write` is the only class needing acknowledgement.
- `409 origin_needs_reconciliation` (with the origin's `details.intent`) when the
origin is parked at a dangling write; resolve the origin first.
- `409 invalid_fork_node` when the origin never entered `from_node`.
- `409 not_a_graph_run` when the run is an ordinary agent run.
- `404 unknown_graph` when the origin's graph is no longer stored (graphs are
in-memory and do not survive a restart); resubmit the identical document, then
fork.
- `dry_run: true` returns `200` with `{ "dry_run": true, "origin": "<id>",
"from_node": "publish", "through_seq": 3, "graph_hash": "sha256:...",
"prefix_event_count": 4, "writes": [ ... ], "unacknowledged_writes": [4],
"would_proceed": false }` and creates nothing. The structural refusals above
still apply under `dry_run` (a fork that could never proceed is reported, not
faked).
### GET /v1/runs/{id}/forks
The forks of a run, as a DERIVED index. The origin is immutable and never points
forward at its children; this answer is a server-side scan of every run's
`forked_from`, labeled `"derived": true` to say so. It is not a fact the origin
recorded.
```json
{ "run": "<id>", "derived": true, "forks": [
{ "run": "<child>", "from_node": "publish", "through_seq": 3, "acknowledged_writes": [4] }
] }
```
- `404 unknown_run` when the id has no history.
### GET /v1/capabilities
What this build of the control plane can do, for a dashboard to probe before
offering a capability-gated action. Additive and honest: a capability is
advertised only when the feature genuinely exists on this server.
The sibling `server` object names the exact build serving the response.
`server.version` is the running binary's own `CARGO_PKG_VERSION`, so it always
agrees with `salvor --version` for the same build. `server.commit` is the
short git hash the build was compiled from, present only when that build had
a `.git` history to read at compile time (a source tarball or a checkout with
no `git` on `PATH` omits the key entirely rather than sending a placeholder
like `"unknown"`), and suffixed `-dirty` when the working tree carried
uncommitted changes at build time.
```json
{
"capabilities": { "fork": true },
"server": { "version": "0.1.0", "commit": "a1b2c3d" }
}
```
A build with no commit information available:
```json
{ "capabilities": { "fork": true }, "server": { "version": "0.1.0" } }
```
### Submitting a graph from the CLI
The `salvor` CLI drives graphs LOCALLY (`salvor graph run <graph.json> --input
<json> [--agent <file> ...]`), the same way `salvor run` drives an agent run; it
has no remote-verb convention, so it does not submit graphs to a server. Submit
and validate over HTTP with `curl` (or an SDK) against the endpoints above.
`salvor fork <run> --from-node <id> --graph <graph.json> [--agent <file> ...]
[--acknowledge-writes <seq,seq|all>] [--dry-run]` is the local flavor of the fork
endpoint: it re-supplies the origin's document (hash-checked against the recorded
one, since a fork reuses the graph unchanged), plans the fork, and drives the
child onward from the fork node, refusing any write the re-walked segment would
re-fire until `--acknowledge-writes` covers it. Same refuse-then-record contract
as the endpoint, exit 1 on an unacknowledged hazard.
## Client-performed tools
### GET /v1/client-tools
Every client-performed tool declaration this server was started with: the
tools an operator declared with `salvor serve --client-tool <FILE>`, which the
client runs itself, in its own process (see `POST
/v1/client-runs/{id}/client-tool-intent` below). This server holds no code for
them; the declaration is name, effect, and schemas, nothing else.
This is how a client-driven loop gets the model's function definitions. A
declaration's `input_schema` IS the tool's parameter schema, the exact one
this server checks a client-tool intent's input against, published here so a
client never keeps a second copy of it that can quietly drift from the one
the server validates against.
Both schemas are checked against a subset of JSON Schema, not the whole of it:
the server honours `type`, `required`, `properties`, `items`, `enum`,
`minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum` (the draft 2019-09
numeric forms, where the keyword's value is the bound), `minLength`,
`maxLength`, `minItems` and `maxItems`, and every other keyword, `pattern` and
`format` among them, is ignored without refusal, so an operator who writes one
has written a comment rather than a rule.
No drive token: this lists server configuration, not run state, so it sits
behind only the bearer-auth layer every other `/v1` route sits behind.
- Response `200`:
```json
{
"client_tools": [
{
"name": "charge_card",
"effect": "write",
"input_schema": { "type": "object", "required": ["amount_cents"], "...": "..." },
"output_schema": { "type": "object", "required": ["charge_id"], "...": "..." },
"trust_completion": true,
"idempotency_key": ["order_id", "amount_cents"]
}
]
}
```
`idempotency_key` is present only when the declaration names key fields, and it
says what this server derives the call's key from (see `client-tool-intent`
below). It is published for the reason `input_schema` is: a client that wants to
derive the key itself, to check this server's work, has to know what the
derivation is over.
`output_schema` is present only when the declaration carries one, following
the zero-vs-absent rule `GET /v1/agents` already applies to `name`: a tool
declared without an output schema cannot be self-completed by a client (see
`client-tool-completion` below), and there is no genuinely empty schema to
fall back to, so the key is omitted entirely rather than sent as `null`.
A server started with no `--client-tool` files answers `{ "client_tools": [] }`,
not an error: nothing declared is a complete, honest state, the same one every
client-tool intent already answers with a clean `unknown_tool`.
Declarations are loaded by the operator and are never registered over HTTP;
see `client-tool-intent` below for why that is not an omission.
## Client-driven runs
Everything above is the server-driven control plane: the server owns the loop
and drives it in a background task. The endpoints in this section are a second,
additive mode that moves ownership of the loop to the client while the server
keeps ownership of the log. The client (a browser folding a run's log in a wasm
`ReplayCursor`, or an SDK) owns the loop and streams the events it produces;
the server owns the durable log
and, on every append, re-folds the log with the pure `salvor-replay`
append-guard to confirm the incoming event is the one legal next event. The two
modes never collide: a client-driven run and a server-driven run cannot share an
id, and each surface serves only its own runs.
A client-driven run may park on a durable timer, and its client is what wakes
it. The generic append accepts the pair (`SleepStarted { wake_at }` and
`SleepCompleted`) for the same reason it accepts `Suspended` and `Resumed`:
both halves are recorded facts the client's own cursor produces, neither
holds a secret, and neither has an effect outside the log. Nothing on the
server waits for the deadline. The wake sweep leaves every client-driven run
alone, current lease or lapsed, because re-driving one from this process
would be a second writer racing the client's drive token for the same
sequence numbers. So the client wakes its own run: on a later drive it
replays its log, finds a `SleepStarted` with no `SleepCompleted` after it,
compares the recorded `wake_at` against a clock reading it records as a
`NowObserved`, and either stops (still asleep, nothing appended) or appends
the `SleepCompleted` and carries on. The server enforces the order of the
pair, refusing a `SleepCompleted` that would close a sleep the log never
started with `409 divergence`; it does not judge the deadline, because
`wake_at` is the client's own recorded instant and the clock that decides it
has arrived is the client's. `GET /v1/runs/{id}` reports such a run as
`{ "state": "sleeping", "wake_at": "<RFC 3339>" }` like any other.
For the same reason, `POST /v1/runs/{id}/resume` (the server-driven surface)
refuses a client-driven run outright, `409 client_driven_run`, whatever its
state: only its own client may drive it, by re-opening `POST /v1/client-runs`.
Both that refusal and the sweep's skip read the run's recorded `driven_by`, not
only the server's in-memory leases, so they hold after a restart and before the
client has re-opened anything.
The generic append carries only the control and deterministic-context events the
client's cursor emits itself, which hold no secret and no side effect:
`RunStarted`, `NowObserved`, `RandomObserved`, `Suspended`, `Resumed`,
`SleepStarted`, `SleepCompleted`, `BudgetExceeded`, `RunCompleted`,
`RunFailed`. The side-effecting steps, which the server must perform because it
holds the key or the binary, have their own endpoints: the model call is the model-step endpoint and the tool call is the
tool-step endpoint below, and a model or tool event is still refused on the
generic append. A step the CLIENT performs in its own process, with its own
secrets, has its own endpoint pair for the same reason: `client-tool-intent`
with `client-tool-completion` for a tool, and `client-model-intent` with
`client-model-completion` for a model call the client made with its own key.
### The drive token
Opening a client-driven run mints a per-run `drive_token`: the single-writer
lease. Every append must present it in the `X-Drive-Token` header. It is the
per-run gate that layers on top of the process-wide bearer, so one authenticated
caller cannot drive another caller's run, and a second live driver without the
current lease is refused.
**A lease is held until it lapses.** Re-opening a run whose driver still holds a
current lease is `409 lease_held`, not a fresh token: the refusal carries
`details.lapses_in_seconds`, the whole seconds until the hold expires if that
driver stays quiet. A driver proves it is alive by presenting its token: every
driving call (append, model step, tool step, an intent, a completion, resolve)
refreshes the lease. Go quiet for the lease TTL (`SALVOR_CLIENT_LEASE_TTL_SECS`,
60s by default) and the hold lapses; the next open takes the run and mints a
fresh lease, and the quiet driver's token is then `403 invalid_drive_token` on
its next call. A `completed`, `failed`, or `abandoned` run is never held: it
takes no more appends from anyone, so it re-opens straight away.
The run's own driver may re-open it, presenting its current token in the
`X-Drive-Token` header. That returns the recorded log under the **same** token,
so a client rebuilding its cursor is not made to give up the lease it holds and
a call already in flight under that token stays valid. `record_prompts` is
ignored on such a re-open; the lease keeps what it was opened with.
**Lapsing is the safety net, not how a drive ends.** A driver that is finished
hands the lease back with `POST /v1/client-runs/{id}/release`, and the run is
free on the very next request instead of a TTL later. Without it a short-lived
process locks out the process that follows it: an SDK invoke returns, the
process exits, and the next one is refused `409 lease_held` for up to a minute
for nothing. A driver that will be busy for longer than the TTL inside one body
(a tool that takes minutes, a model stream it is rendering itself) makes no
drive call while it works, so it says "still here" with `POST
/v1/client-runs/{id}/heartbeat` and keeps the run it never left.
**A resolve clears the lease.** Recording a dangling write by hand through `POST
/v1/runs/{id}/resolve` says the driver that opened that write is gone: it never
came back to record what the write did, and the caller unsticking it presents no
drive token, so it is somebody else. The lease that dead driver left behind is
dropped along with the resolution, and the client re-opens the run straight away rather than being
refused for the rest of the TTL right after being told the run is unstuck. The
client's own `POST /v1/client-runs/{id}/resolve` is the exception: it presented
the run's current token to get in, which is the driver saying it is right here,
so it keeps the lease and carries on.
`salvor resolve` on the command line is a third case. The CLI writes the store
directly and has no way to reach a running server's memory, so a lease held by a
live server survives a CLI resolve and lapses on its own; a client re-opening
after one waits at most the TTL. Use the HTTP resolve against a live server to
free the run at once.
The rule is not "newest caller wins", and deliberately so. The caller re-opening
a live run is usually not the driver that has it, but a second app instance, a
duplicated tab, or a middleware retrying after a failed call. Handing it the
lease puts two drivers on one log: both run the same step, and the one that
loses the race to a position takes a `409 divergence` after doing the work. A
refreshed tab still resumes, because the tab that went away stopped presenting
its token and its lease lapses.
Leases live in the server process and do not survive its restart, but the run
does. A client-driven run's `RunStarted` records `driven_by: "client"`, stamped
by the server when it accepted that event, and that is what a restarted server
reads to know the run is a client's to drive. So a restart does not strand a
run: re-opening it returns its recorded log with a fresh lease (see below), and
until then the surfaces that would otherwise become a second writer still
refuse it (`POST /v1/runs/{id}/resume` answers `409 client_driven_run`, and the
wake sweep leaves its due timer alone). Every lease minted before the restart is
gone, so a client that was mid-drive re-opens the run to get a current one.
### POST /v1/client-runs
Open a fresh client-driven run, or re-open (resume) one this server holds.
- Request:
```json
{ "agent": "sha256:34e0...", "input": <any json>, "run_id": "<uuid, optional>",
"record_prompts": false }
```
`run_id` is optional; when omitted the server mints one. `agent` and `input` are
accepted for forward compatibility with the server-performed model step; this
endpoint records them nowhere, because the client appends its own
`RunStarted` (carrying the agent hash and input) as the run's first event.
`record_prompts` is stored against the run for the server-performed step.
- Response `201` for a fresh run:
```json
{ "run": "6f...", "drive_token": "dt_...", "log": [] }
```
The empty `log` is what the client builds its cursor from. The client then
appends its own `RunStarted` at seq 0 through the events endpoint.
- Response `200` for a re-open, with `log` carrying every recorded envelope and
a `drive_token`: a fresh one when the run was unheld, or the caller's own one
back when it presented it (see [The drive token](#the-drive-token)). Two
things make a `run_id` re-openable, and either is enough: this server opened
it since it started (it holds the lease), or the run's own log says it is
client-driven, its `RunStarted` carrying `driven_by: "client"`. The second is
what makes a restart survivable: the lease registry dies with the process, the
log does not, so a server that has just come back re-opens a run its client is
still driving instead of refusing it as foreign. Adopting a run this way mints
a fresh lease, and the returned `log` is the recorded state the client rebuilds
its cursor from, dangling intent and all.
- `409 lease_held` when another driver's lease on the run is still current and
the request did not present that lease's token. Nothing is recorded and no
lease is minted, so the driver that has the run keeps it.
```json
{
"error": {
"code": "lease_held",
"message": "another driver holds run 6f...; its lease lapses in 60s if that driver goes quiet, and re-opening works then (or as soon as the run finishes)",
"details": { "lapses_in_seconds": 60 }
}
}
```
A caller that means to take over waits `lapses_in_seconds` and asks again. A
caller that IS the driver presents its token instead and is not refused. The
hold never survives the server process: a restart leaves no lease to hold,
so the first open after one always succeeds.
- `409 run_exists` when the chosen `run_id` already has history and its log does
not record it as client-driven, which means it is a server-driven run. That
refusal is unchanged by a restart, so the two modes still cannot collide over
one store.
### POST /v1/client-runs/{id}/release
Hand the drive-token lease back. Requires the `X-Drive-Token` header. No request
body is read.
- Response `200`:
```json
{ "released": true }
```
- Response `200` with `"released": false` when the run has no lease here: it was
released already, it lapsed, or this server never opened the run. The call is
idempotent, and finding nothing to give back is not an error, because the
caller's goal (a run nobody is holding) is already true.
- `403 invalid_drive_token` when a lease stands on the run and the request did
not present its token, a missing token included. Unlike the driving endpoints,
a missing token here is not `401 missing_drive_token`: the question a release
asks is not "did you bring credentials" but "is this hold yours to end", and
the answer is no either way. Nothing is dropped.
Only the lease goes. The run's log is untouched and it stays a client-driven run
(its `RunStarted` still carries `driven_by: "client"`), so the next open adopts
it exactly as it would after a restart, `POST /v1/runs/{id}/resume` still answers
`409 client_driven_run`, and the wake sweeper still leaves a due timer to its
client. All three read the log marker, not the lease.
An SDK calls this when an invoke ends, in the success path and the error path
alike. It is the difference between the next process picking the run up
immediately and being refused for the rest of the TTL.
### POST /v1/client-runs/{id}/heartbeat
Refresh the lease without driving the run. Requires the `X-Drive-Token` header.
No request body is read.
- Response `200`:
```json
{ "lapses_in_seconds": 60 }
```
`lapses_in_seconds` is the whole lease TTL as of this beat (rounded up to whole
seconds, never below `1`), so a driver picks its interval from the answer rather
than being told the server's configuration some other way. Beat comfortably
inside it.
- `401 missing_drive_token` / `403 invalid_drive_token` on a missing or
superseded lease; `404 unknown_run` when the id is not a client-driven run
this server opened.
Presenting the drive token has always been the heartbeat, and every driving call
carries it. What that misses is the driver that makes no drive call for longer
than the TTL because it is inside one long body: a tool that takes minutes, a
model call it is streaming to its own screen. Its lease would lapse mid-body and
another opener could take a run whose driver never went anywhere. Re-opening
under the held token has the same effect (it keeps the lease and counts as proof
of life), but it re-reads the whole recorded log every beat to say four words;
this is the call the SDKs use.
### GET /v1/client-runs/{id}/log
The recorded envelopes, for a refreshed tab to rebuild its cursor.
```json
{ "log": [ <envelope>, ... ] }
```
Each envelope is exactly the pinned event-envelope wire JSON the event stream
and `salvor history --json` use. `?from_seq=<n>` returns only envelopes at or
after `n`, so a client that already holds a prefix fetches just the tail. The
read needs no drive token, and it needs no lease either: it serves any run
that is client-driven, whether that is known from this server's lease
registry or from the recorded log's own `driven_by: client` marker, so it
still answers after the driver released the lease, the lease lapsed, or the
process restarted. A server-driven (or unknown) run is `404 unknown_run`.
### POST /v1/client-runs/{id}/events
The generic guarded append. Requires the `X-Drive-Token` header.
- Request:
```json
{ "events": [ <EventEnvelope>, ... ] }
```
- Response `200`:
```json
{ "appended": [ <seq>, ... ] }
```
The server re-folds and appends the batch in order. The whole batch is validated
before anything is written, so a batch that turns illegal appends nothing.
A `RunStarted` event in the batch may carry `labels` (the client builds this
event itself; see [POST /v1/client-runs](#post-v1client-runs) above). The same
bounds `POST /v1/runs` enforces apply here, at the one point this server ever
sees them for a client-driven run: at most 16 labels, each key at most 64
bytes, each value at most 256 bytes. A `RunStarted` carrying labels over the
bounds is `400 bad_request`, and nothing in the batch is written.
A `RunStarted` event in the batch is also **stamped** with
`driven_by: "client"` before it is folded or written, whatever the submitted
event carried in that field. Reaching this endpoint means holding the run's
lease, so the run is client-driven, and its head is where that fact is recorded
durably; a caller cannot set the marker anywhere else. The stamp happens before
the retry comparison below, so re-appending the same `RunStarted` bytes is
still the byte-identical no-op it was.
Every envelope's `recorded_at` is **server-stamped**: the server overwrites it
with its own clock reading before folding or writing the event, regardless of
what the submitted envelope carries in that field. `recorded_at` is a required
field on the wire (the pinned `EventEnvelope` shape every event stream and
`salvor history --json` share), so a client must still send one, but its value
is ignored entirely: a client may send the current time, the Unix epoch, or
anything else, and the server's stamp always wins. This keeps `recorded_at`
meaning "when this store durably recorded the event" rather than "whatever a
browser's clock happened to read," uniformly with the model-step and tool-step
endpoints below, which have always stamped their own intents and completions
this way.
Semantics, keyed by sequence number:
- A byte-identical re-append at an already-recorded seq is a `200` no-op (the
retry-safe case: a tab resends after a network blip). Its seq is still
reported in `appended`, and the log does not grow.
- Different bytes at an already-recorded seq is `409 divergence`.
- An illegal next event (a wrong sequence number, a completion that does not
correlate to its intent, a second pending intent, an event after a terminal
event, or a malformed head) is `409 divergence`, with the append-guard's
precise reason in `message`.
- A model or tool event is `422 unsupported_event_kind`: those are recorded
through the model-step and tool-step endpoints, or, for a call the client
performs in its own process, through the client-tool-intent and
client-tool-completion endpoints. A client-performed tool call is possible; it
is just not possible by hand-appending an event, because the effect class, the
input check, and the idempotency key are the server's to decide from an
operator's declaration.
- A missing drive token is `401 missing_drive_token`; a token that is not the
run's current lease is `403 invalid_drive_token`.
- A body over the `8 MB` cap, or a batch over 1024 events, is `413
payload_too_large`.
### POST /v1/client-runs/{id}/model-step
The server-performed model call. The client owns the loop and decides when to
call the model and with what; the server performs the call (it holds the key)
and records it. Requires the `X-Drive-Token` header.
- Request:
```json
{ "seq": 3, "request": <MessageRequest as JSON> }
```
`seq` is the log position the client's cursor reserved for the model intent.
`request` is the client's canonical model request value. The server recomputes
`request_hash` from `request` with the same canonical hash the runtime uses, so
the client cannot record a hash that does not match what was sent.
- Response `200`:
```json
{ "response": <MessageResponse as JSON>, "usage": { "input_tokens": 10, "output_tokens": 5 } }
```
The server appends `ModelCallRequested { seq, request_hash, request_body? }`
write-ahead (the body is recorded only when the run was opened with
`record_prompts: true`), performs the call through the injected model executor,
appends `ModelCallCompleted { seq, response, usage }`, and returns the
completion. The client feeds the response and the hash back to its cursor, which
advances over the two now-recorded events.
Retry identity is `(seq, request_hash)`, mirroring `ReplayCursor::model_call`:
- A step already completed at `seq` with the same hash returns the recorded
completion; the provider is not called again and the log does not grow. This
is the no-re-pay case.
- A dangling intent at `seq` with the same hash (the tab died mid-call) is
re-executed: an unanswered model request has no external effect to double, so
the fresh completion correlates to the recorded intent.
- A different hash at `seq`, or a non-model event there, is `409 divergence`. A
`seq` beyond the log's end is `409 divergence` too.
The model executor is a general injection seam the embedding binary supplies
(the `AgentFactory` pattern): `salvor serve` wires a default from its own model
client out of the box, and another host injects its own. The default executor
reads `ANTHROPIC_API_KEY` for its credential and targets the public endpoint;
setting `SALVOR_MODEL_BASE_URL` points it at a local or offline endpoint
speaking the same Messages wire protocol instead. With no key set, no auth
header is sent at all, which is what local endpoints expect. A step against a
server with no executor wired is `503 model_executor_unavailable`, and no intent
is written for the call it cannot make, so the run stays drivable once one
exists. A provider failure is `502 model_execution`; no completion is recorded,
so the write-ahead intent is left dangling (the legal crash story) and a retry
re-issues the call safely.
#### Streaming variant
With `Accept: text/event-stream` (or `?stream=1`) the response is a server-sent
event stream for a live ticker:
```text
event: delta
data: { "type": "text_delta", "index": 0, "text": "the plan: " }
event: complete
data: { "response": <MessageResponse as JSON>, "usage": { ... } }
```
Each provider event that carries ticker text (text and thinking deltas, and the
final usage) rides a `delta` frame while the call runs; the assembled completion
is recorded once at the end and carried on the closing `complete` frame. The
recorded `ModelCallCompleted` is byte-identical to the non-streaming path for the
same underlying response. A tab that drops mid-stream leaves a dangling intent,
re-issued safely on resume; a mid-stream provider failure sends an `error` frame
and records no completion. A model step that resolves to a replay (already
recorded) streams a single `complete` frame carrying the recorded completion.
### POST /v1/client-runs/{id}/tool-step
The server-performed tool call. The client owns the loop and decides when to call
a tool and with what; the server performs the call (it holds the binary or the
credential the tool needs) and records it. Requires the `X-Drive-Token` header.
- Request:
```json
{ "seq": 5, "tool": "render", "input": <any json>, "idempotency_key": null }
```
`seq` is the log position the client's cursor reserved for the tool intent.
`tool` names a tool the server's registry holds. `input` is the tool's typed
input, recorded on the intent verbatim. `idempotency_key` is optional; for an
`Idempotent` tool the client draws it from a recorded `RandomObserved` so it
reproduces on replay. A client-declared `effect` field is accepted for shape
parity but ignored: the recorded effect is the tool's operator-declared one, so a
caller cannot up- or down-grade it.
- Response `200`:
```json
{ "output": <the json the tool returned> }
```
The server takes the effect from the registration, appends `ToolCallRequested {
seq, tool, input, effect, idempotency_key }` write-ahead, dispatches the tool,
appends `ToolCallCompleted { seq, output }`, and returns the output. The client
feeds the output back to its cursor, which advances over the two now-recorded
events.
Retry follows the effect table, mirroring `ReplayCursor::tool_call`:
- A step already completed at `seq` with the same `(tool, input, effect, key)`
returns the recorded output; the tool is not dispatched again and the log does
not grow. This is the no-re-execution case.
- A dangling `Read` or `Idempotent` intent at `seq` (the tab died mid-call) is
re-executed under the RECORDED idempotency key, so an idempotent retry reuses
the exact key the provider collapses duplicates on. The fresh completion
correlates to the recorded intent.
- A dangling `Write` intent at `seq` is `409 needs_reconciliation` carrying the
recorded intent in `details.intent`, and the tool is not dispatched: the write
may have landed, and only the resolve endpoint below may record its completion.
- A different `(tool, input, effect, key)` at `seq`, or a non-tool event there,
is `409 divergence`. A `seq` beyond the log's end is `409 divergence` too.
The tool registry is a general injection seam the embedding binary supplies (the
same pattern as the model executor and the `AgentFactory`): the binary registers
named tools whose effects it declares. `salvor serve` wires an empty registry, so
every tool-step is `404 unknown_tool` until a host registers a tool; another host
(for example a render server) registers its tools. A step naming an unregistered
tool is `404 unknown_tool`, and a step against a server with no registry at all
is `503 tool_registry_unavailable`; in both cases no intent is written, so the
step is retriable once the tool is present. A dispatch failure is `502
tool_execution`; no completion is recorded, so the write-ahead intent is left
dangling (the legal crash story), drivable-or-reconcilable per the tool's effect.
The CLI's `salvor serve --demo-tools` is the one built-in exception, off by
default: it registers three deterministic demo tools (`lookup_invoice` read,
`issue_refund` write, `send_email` idempotent, see
`salvor_cli::demo_tools`) so a demo or the served end-to-end suite can run a
tool-bearing graph with no embedding host at all. It changes nothing about the
seam above: the demo tools register through the exact same `ToolRegistry`
any other host would, and a plain `salvor serve` with no flag still wires it
empty, byte for byte.
### POST /v1/client-runs/{id}/client-tool-intent
Open a tool call the CLIENT performs, in its own process, with its own secrets.
This server holds no code for such a tool and never dispatches it; it records
that the call was asked for, before the effect happens, exactly as the
write-ahead rule demands of a call it performs itself. Requires the
`X-Drive-Token` header.
- Request:
```json
{ "seq": 5, "tool": "charge_card", "input": <any json> }
```
- Response `200`, a call that still has to be performed:
```json
{ "seq": 5, "idempotency_key": "sha256:...", "effect": "write", "settled": false }
```
- Response `200`, a call whose completion is already recorded:
```json
{ "seq": 5, "idempotency_key": "sha256:...", "effect": "write", "settled": true,
"output": <the recorded output> }
```
Notice what the request does NOT carry, compared with `tool-step` above: no
`effect` and no `idempotency_key`.
`settled` is `true` when the intent at this position already has its
completion recorded, `false` otherwise, and a settled answer carries that
completion's `output`. It matters most to a caller re-posting an intent it
believes it already opened: a payments caller retrying after a dropped response
gets back the same `200` and the same key either way, and without `settled` it
cannot tell "safe to perform the call" from "already performed and completed, do
not do it again" without separately reading the log. `output` is omitted while
the call is open.
The effect is the operator's, from the declaration, for the same reason
`tool-step` refuses a client-declared effect: a caller must not be able to
up- or down-grade its own write into a freely retried read.
The idempotency key is DERIVED by the server, and this deliberately differs from
`tool-step`, where the client supplies it. There, salvor performs the call, so
the party choosing the key is not the party making the write. Here the client
both chooses and performs, which is the one case where the party choosing the
key is also the party who benefits from a duplicate landing. Deriving it removes
the choice. What the key is derived FROM is the operator's, from the
declaration's `idempotency_key`:
- **No fields declared** (the default): a canonical hash of `{ run, seq, tool }`.
The call's position in the run. This is an attempt identifier and promises one
thing: the same position, retried, presents the same key, so an honest retry
after a dropped response presents the identical key and the provider collapses
the pair. Two calls at two positions are two calls, however alike their
arguments.
- **Fields declared**, `idempotency_key = ["order_id", "amount_cents"]`: a
canonical hash of `{ run, tool, order_id, amount_cents }`, with the values
taken from this intent's input. `seq` is deliberately absent, which is the
whole difference: the same refund asked for twice in one run derives one key
both times. The order the fields are named in does not change the hash, since
the canonical form sorts object keys.
The run id is on both shapes, so a declared key is an identity within one run and
never collides with another run's. A client can derive either key independently,
with any canonical-JSON SHA-256, to check the server's work.
**A declared key deduplicates.** For a `write` or an `idempotent` call carrying a
declared key, the server claims the identity in the store before the intent is
written, exactly as the runtime does for a tool that declares its own key. A
second intent whose key fields match a call this run already completed does not
open new work: the completion is copied to the new position, carrying a
`deduplicated_from` naming what it copied, and the response comes back
`settled: true` with that output. **The client performs nothing.** A `read`
never deduplicates, whatever it declares: a read has no effect worth an identity,
and answering a repeated read from an older call would freeze a loop that is
polling for a change on purpose.
The input is validated against the declaration's `input_schema` before anything
is written, so a malformed call never becomes history. The intent then goes
through the same append-guard every other write on this surface uses.
Refusals, each of which writes nothing:
- an undeclared tool is `404 unknown_tool`;
- an input failing `input_schema` is `400 bad_request`;
- an input missing a field the declared `idempotency_key` derives from is `400
bad_request`, naming the field;
- a `seq` the log is not ready for, or a different event already recorded
there, is `409 divergence`;
- a declared key held by a call that is open and unfinished is `409 divergence`.
Nothing is recorded, so the call can simply be opened again once the holder is
settled.
A byte-identical re-post at an already-recorded position is a `200` that
re-derives the same key and writes nothing: the safe retry a dropped response
leaves behind.
Declarations are loaded by the operator (`salvor serve --client-tool <FILE>`, or
`AppState::with_client_tools` for an embedding host) and are NEVER registered
over HTTP. That is not an omission. A declaration fixes the effect class, so a
client able to POST one would be deciding whether its own writes are subject to
the write-ahead rule at all.
### POST /v1/client-runs/{id}/client-tool-completion
Record what a client-performed tool call returned. The client ran the call;
salvor did not witness it, so everything this endpoint can check, it checks
before the report becomes history. Requires the `X-Drive-Token` header.
- Request, a call that returned something:
```json
{ "seq": 5, "output": <the json the client says the call returned> }
```
- Request, a call that returned nothing because it failed:
```json
{ "seq": 5, "error": { "message": "the provider timed out", "kind": "handler" } }
```
- Response `200`, for either:
```json
{ "seq": 5, "completed": true }
```
Exactly one of `output` and `error`. A body carrying both, or neither, is `400
bad_request`: they say opposite things about the same call and this server has no
way to pick between them.
**A failure is a completion, not a new state.** A reported `error` records the
same `__salvor_error` sentinel output the runtime records when a NATIVE tool
exhausts its retries, byte for byte, so a log written through this endpoint means
to a replay exactly what a natively recorded one means: the call is closed, the
run carries on, and a later replay reads the failure back rather than performing
the call again. The recorded output is
```json
{"__salvor_error": {"is_error": true, "kind": "handler",
"message": "the provider timed out", "attempts": 1}}
```
`message` is recorded verbatim, in full. `kind` is optional on the wire and is
one of `invalid_input`, `handler`, or `output_serialization`, the dispatch layer
that failed; absent means `handler`, which is what a client tool that ran and
threw is. Any other value is `400 bad_request`. `attempts` is not on the wire and
is always `1`: it counts executions inside salvor's own retry loop, and salvor
ran no loop over a call it did not dispatch.
The declared `output_schema` and every `require_equal` field are skipped on the
`error` shape, because there is no reported value for either to look at. The
trust rules are not: see the refusals below.
Refusals, each of which records nothing:
- the log does not end at a tool intent, or ends at one whose `seq` is not the
one named: `409 divergence`;
- the pending intent was performed by the SERVER: `403
client_completion_refused`. A client must not close a call salvor made, since
salvor holds the real result;
- the declaration says `trust_completion = false`: `403
client_completion_refused`. This holds for a reported failure too. "It did not
land" is a claim about money made by the party that benefits from it being
believed, so an untrusted write is left dangling for a person exactly as an
untrusted result is;
- the declaration carries no `output_schema` **and** the body reports an
`output`: `403 client_completion_refused`. With nothing to check the report
against, the completion is unfalsifiable, which is exactly what the schema
exists to prevent. A reported `error` is unaffected, since it carries no value
to check;
- the output fails the declared `output_schema`: `400 bad_request`.
A refusal is not a dead end, and it needed no new run state. The log still ends
at the recorded `ToolCallRequested`, and for an `Effect::Write` the pure fold in
`salvor-replay` already reports that as `needs_reconciliation`, because an
uncompleted write intent as the log's last word is precisely what that status
means. The resolve endpoint below already exists to settle it by hand, once a
person has verified externally whether the call landed. So `trust_completion =
false` is enforced here, at the completion boundary, and deliberately not in
`derive_state`: that fold is a pure function of the log with no access to
declarations, and it must stay that way, because a log has to mean the same
thing to a replay on a machine that has never seen this server's declaration
files.
The recorded `ToolCallRequested` carries `performed_by: "client"`, which is how
a later reader tells a call salvor witnessed from a call it was told about. A
server-performed intent omits the field entirely.
### POST /v1/client-runs/{id}/client-model-intent
Open a model call the CLIENT performs, in its own process, with its own key and
its own model configuration. This server does not make the call and never sees
the request; it records that the call was asked for, before it happens, exactly
as the write-ahead rule demands of a call it performs itself. Requires the
`X-Drive-Token` header.
- Request:
```json
{ "seq": 3, "request_hash": "sha256:...", "request_body": <the request, optional> }
```
- Response `200`, an intent that still has to be performed:
```json
{ "seq": 3, "settled": false }
```
- Response `200`, an intent whose completion is already recorded:
```json
{ "seq": 3, "settled": true,
"response": <the recorded response>,
"usage": { "input_tokens": 10, "output_tokens": 5 } }
```
Notice what the request does NOT carry, compared with `model-step` above: no
`request`. `model-step` recomputes `request_hash` from the request body it is
handed, so a client cannot record a hash that does not match what was sent.
Here it can. The request never reaches this server, because this server is not
the one sending it, so the hash is the client's claim over its own request, and
the response reported later is the client's claim about what came back. Salvor
did not witness the call; it is trusting the report, exactly as it trusts a
client-performed tool result.
What the trust buys is the point of the endpoint: a resume replays the recorded
answer instead of paying the provider for it a second time. The claim is also
self-punishing rather than dangerous to anyone else, because the hash is a key
into this run's own log: a client that hashes inconsistently diverges against
its own history and nobody else's.
`settled` is what a middleware short-circuits on. `false` means the call still
has to be made; `true` means it is already recorded, and the recorded
`response` and `usage` ride along so the caller can return them without a
second request and without a separate log read.
`request_body` is recorded on the intent only when the run was opened with
`record_prompts: true`, the same rule the server-performed step reads off the
run's lease. Sent to a run that does not record prompts, it is dropped and
never written. It is informational either way: correlation is on `request_hash`
alone.
Refusals, each of which writes nothing:
- a different `request_hash` at an already-recorded position: `409 divergence`;
- a non-model event at that position: `409 divergence`;
- an intent at that position that this SERVER performed: `409 divergence`. The
client's cursor and the log disagree about who owns that step;
- a `seq` the log is not ready for: `409 divergence`.
A re-post at a recorded position with the same hash is a `200` that writes
nothing: the safe retry a dropped response leaves behind, and the replay a
later drive is built on.
The recorded `ModelCallRequested` carries `performed_by: "client"`, which is how
a later reader tells a call salvor witnessed from a call it was told about. A
server-performed intent omits the field entirely. The fold reads no performer at
all, so a client-performed call moves a run exactly as a server-performed one
does: `awaiting_model` while the intent is open, and its tokens counted toward
every budget once the completion lands.
For the same reason in the other direction, `model-step` refuses a position
holding a client-performed intent with `409 divergence`, and calls no provider:
performing it there would let this server witness a response for a call the log
attributes to the client.
### POST /v1/client-runs/{id}/client-model-completion
Record what a client-performed model call returned. Requires the
`X-Drive-Token` header.
- Request:
```json
{ "seq": 3, "response": <what the client says the provider returned>,
"usage": { "input_tokens": 10, "output_tokens": 5 } }
```
- Response `200`:
```json
{ "seq": 3, "completed": true }
```
`usage` is required, not optional: it is what a token budget counts, and a
completion that quietly reported none would under-count every budget the run is
held to.
Refusals, each of which records nothing:
- the log does not end at a model intent, or ends at one whose `seq` is not the
one named: `409 divergence`;
- the pending intent was performed by the SERVER: `403
client_completion_refused`. A client must not close a call salvor made, since
salvor holds the real response.
That is the whole list, and it is shorter than `client-tool-completion`'s on
purpose. The tool completion's remaining refusals all come from the operator's
declaration (`trust_completion`, `output_schema`, `require_equal`), and a model
call has no such declaration to check against: its response shape is the
provider's, not an operator's. The response is recorded verbatim, and the
recorded `ModelCallCompleted` is byte-identical to the one the server-performed
step writes for the same response.
### POST /v1/client-runs/{id}/resolve
Record the completion of a dangling write by hand for a client-driven run, the
drive-token-gated twin of the server-driven `POST /v1/runs/{id}/resolve`.
Requires the `X-Drive-Token` header.
- Request:
```json
{ "output": <the json the tool returned> }
```
- Response `200`:
```json
{ "run": "6f...", "resolved": true }
```
State-validated exactly like the server-driven resolve: it is legal only when the
run's log ends at a dangling `Write` intent, it correlates the caller-supplied
output to that intent, and it dispatches nothing. After it records the completion
the run is drivable again, so the client re-fetches the log and its cursor sails
past the once-dangling intent.
Declaration-validated exactly like it too, through the same code: a client
performed call's output is checked against the declaration's `output_schema` and
its `require_equal` fields before anything is written, and an undeclared tool is
refused. See `POST /v1/runs/{id}/resolve` above for the full list and the
reasoning. The completion this endpoint records carries `settled_by: "operator"`
for the same reason.
**This one keeps the caller's lease**, unlike `POST /v1/runs/{id}/resolve`, which
drops it. Getting in here at all means presenting the run's current drive token,
which is the driver saying it is right here, so there is no dead lease to clear
and revoking a live one would strand the very caller that just proved it is
alive.
- `409 wrong_state` when the run does not need reconciliation (there is no
dangling write to resolve). The message says what the unfinished call needs
instead, and never names `recover`.
- `400 bad_request` when the output does not satisfy the declaration for a
client-performed call, or the tool is no longer declared here.
- `401 missing_drive_token` / `403 invalid_drive_token` on a missing or superseded
lease; `404 unknown_run` when the id is not a client-driven run this server
opened.
## Driving a run
Starting or resuming a run means model and tool calls, which are long, so the
handlers do the fast synchronous part (validate, refuse a bad state, mint or
check the id) and hand the run to a background task that drives it to its next
resting point. The handler returns the run id immediately.
The run is designed to outlive its request. Every event is
persisted to the store inside the driving task, before the task moves on; the
task holds no state the store does not already have. So aborting the task or
dropping the whole server mid-run loses nothing: a fresh server over the same
store recovers the run from its log and continues it, re-executing no completed
model or tool call. That is the same durability the CLI has, over HTTP, and it
is exercised by the kill-safety test.
### Waking a sleeping run
A run parked on a durable timer (`{"state":"sleeping","wake_at":...}`) is
passive data. Nothing in the server holds it and nothing fires at its instant;
it continues only when something re-drives it, at which point the runtime reads
the clock and either records the wake or leaves the run asleep. Driving a run
early therefore cannot wake it: the deadline is enforced inside the run, not by
whoever asked.
`salvor serve` runs a **wake sweeper** for this. On an interval it lists the
runs whose recorded `wake_at` is at or before now and re-drives each one through
the same path `POST /v1/runs/{id}/resume` takes for a recoverable run, so a
woken run behaves identically whether a person or the clock woke it.
- **On by default**, every `--wake-interval` seconds (default `60`). A server
that held a store and let its timers pass would be silently wrong, so this is
not an opt-in.
- **`--wake-interval 0` turns it off**, for an operator who sweeps from cron
with `salvor wake` instead and does not want two things reaching for the same
run.
- **It never fights a driver already running.** A run a task in this process is
still driving is skipped, and the sweep drives sequentially, so no run is
driven twice at once.
- **What this server holds decides it, not what started the run.** By the
hash a run recorded: an agent run wakes once that agent is registered with
`POST /v1/agents`, MCP tools and all, because the server rebuilds the agent
from that same definition; a graph run wakes once its document is
submitted with `POST /v1/graphs` and every `tool` node it carries names a
tool this server's own registry holds (empty by default). Over HTTP a
`tool` node resolves only against that registry, never against tools an
agent's own MCP declarations reach, so a graph run built that way cannot be
woken here regardless of what is registered. Two things leave a run
asleep: its recorded hash is not registered here at all (typically a run
started from the CLI against a store this server never saw), or it is a
graph run with an unmet `tool` node. Each case is logged and skipped, once
per sweep; the run stays due, so an operator wakes it instead with `salvor
wake`, passing the same `--agent`/`--graph` files the run needs.
- **One bad run does not stop the sweep.** Every failure is per-run, and the
loop carries on to the next.
There is no wake endpoint. Waking is a re-drive, and `POST /v1/runs/{id}/resume`
already is one: sending it to a sleeping run whose deadline has passed wakes it.
Sending it before the deadline is refused `409 still_sleeping` carrying
`wake_at` and `remaining_seconds`, because the drive would record nothing and a
`202 driving` for a run that did not move is worse than a refusal that says
when to come back.
### A tool can start the timer
A tool parks its own run by returning the sleep outcome, which the runtime
records **inside** that call's `ToolCallCompleted` (as
`{"__salvor_sleep": {"wake_at": "..."}}`) before appending `SleepStarted`. The
recorded order is therefore intent, completion, `SleepStarted`.
That `{"__salvor_sleep": ...}` shape is what the runtime writes into the log
from a native tool's `ToolOutcome::Sleep`; it is not a value a tool server
sends.
An MCP server can park the run that called it by putting the request under
`_meta` on its tool result, in the `salvor` namespace:
`{"_meta": {"salvor": {"suspend": {"reason": "...", "input_schema": {...}, "kind": "signal"}}}}`
to wait for an input, or
`{"_meta": {"salvor": {"sleep_until": "2026-08-14T09:00:00Z"}}}` to wait
until an instant. `_meta` is the extension point the MCP specification
reserves on every result, so a host that is not salvor reads an ordinary
result with an unfamiliar metadata key. `kind` is optional and its only
value is `"signal"`, meaning an external system owes the run a payload; omit
it and the run waits on a person. `reason` and `input_schema` are both
required. `sleep_until` is an RFC 3339 instant, never a duration: the
runtime records the instant and replay reproduces it. The recorded order is
intent, then the tool call's completion, then `SleepStarted` or `Suspended`;
the completion settles the call and releases its idempotency claim before
the wait begins, so a run parked for a week blocks no other run and holds no
MCP process. A request that is malformed, names both keys, sits on a result
flagged `isError`, or uses a key salvor does not know fails the tool call
with a message naming `_meta.salvor` and the problem; it is never passed
through as ordinary output. That failure is never retried, on any effect
class: the request is already in hand, so reading it a second time reaches
the same refusal, and the tool executes exactly once. A result with no
`_meta.salvor` records exactly as it always has.
That order is the point. The completion settles the call, and for a call
carrying an idempotency key it settles the store's claim in the same atomic
append, so a run that sleeps for a week holds no claim while it sleeps and a
second run under the same key is never told `CallInFlight` by a sleeper. It
also means a process death during the sleep leaves no dangling write intent:
the write already completed.