Rusty Worker
Worker-side SDK for rusty-agent-runtime remote node
execution: one Node trait, remote impls behind the same trait. A worker is
an HTTP service that hosts Node handlers by name; a graph node registered as
RemoteNode calls into it transparently.
Endpoints
POST /execute— accepts a JSONNodeTask, dispatches to the handler registered underNodeTask::node, and replies with a JSONNodeTaskResponse:Ok(output)→{ "output": ... }Err(interrupt)→{ "interrupt": <value> }(HITL across the wire)Err(e)→{ "error": "<message>" }
GET /ok— liveness + capability probe: protocol version and the registered handler names (sorted).
Status codes:
200 OKfor all handler-level outcomes (success, handler error, interrupt, unknown handler, handler panic) — the outcome lives in the body, soRemoteNodenever mistakes a worker-side application error for a transport failure. Handler panics are caught and returned as an error body: a dropped connection would read as a transport failure client-side and be retried, silently replaying node logic.400 Bad Requestwhen the protocol version is unsupported — a client/worker mismatch the client treats as fatal (never retried).
Registering handlers
WorkerRegistry::register accepts anything that implements Node —
which, thanks to the blanket impl in the core crate, includes ordinary async
closures Fn(NodeContext) -> impl Future<Output = Result<NodeOutput>>,
named Node impls, and Arc<dyn Node>. The ergonomics match
GraphBuilder::add_node exactly; registering the same name twice replaces
the previous handler.
use *;
use ;
# async
On the graph side, point a RemoteNode at the same handler name:
builder.add_node;
Error semantics across the wire
Handler errors are flattened to a message string in NodeTaskResponse::error
and arrive client-side as RustyError::Node, which the executor treats
as a hard failure — the retryable classes (Llm, Tool) do not survive
the wire. A remote node whose transient failures should be retried must rely
on transport-level retry (connection/timeout/5xx on the client) or surface
retryable outcomes through its own protocol on top of the extra config
channel.
Durable activities (R0.6): ActivityWorker
ActivityWorker is the pull-based counterpart to serve: instead of
answering one-shot /execute calls, it claims leased tasks from the
rusty-agent-server task queue, executes the Activity registered for the task's
kind while a background heartbeat renews the lease, and settles the task
with a complete or fail call. The server re-queues tasks whose lease
expires, so a crashed worker never strands work — the promise is
effectively-once execution when activity side effects are idempotent, not
universal exactly-once. (The full design lives in
docs/durable-work-design.md; the failure
taxonomy is the frozen rusty_agent_runtime::durable::ErrorClass.)
A task is a generic unit of durable work — a kind label plus an arbitrary
JSON payload. The handler receives an ActivityContext (task_id,
attempt, idempotency_key, payload) and returns the JSON value stored
as the task's result.
Protocol (client side of the server task endpoints):
| Call | Body | Success | Meaning of 409 |
|---|---|---|---|
POST {base}/tasks/claim |
{worker_id, pools?, lease_ms} |
200 {"task": {…}} (the task record); 204 = no work |
— |
POST {base}/tasks/{id}/heartbeat |
{worker_id, lease_ms} (every lease / 3) |
200 {lease_expires_at} |
lease lost → abort the activity, no settle call |
POST {base}/tasks/{id}/complete |
{worker_id, result} (any JSON) |
200 with the updated task record |
task already lost/settled → outcome dropped |
POST {base}/tasks/{id}/fail |
{worker_id, error_class, message, retryable} |
200 {requeued, next_attempt_at, dead} |
task already lost/settled → outcome dropped |
Semantics:
- One activity in flight per worker loop; run several
ActivityWorkerinstances for parallelism and route them withwith_pools. - Lease loss aborts execution: on a heartbeat
409the handler future is aborted (dropped at its next yield point, via aCancellationToken) and the worker returns to claiming without settling — the server's reassignment is authoritative. Handlers must tolerate being abandoned mid-effect and should key external side effects byActivityContext::task_id()/idempotency_key()so redelivery is effectively-once. - Graceful drain (R0.6 wave 2c): cancelling the shutdown
CancellationTokenis the drain request — idempotent, and race-safe against an in-flight claim poll. Draining stops claiming immediately; an in-flight activity keeps heartbeating and settles normally, bounded bywith_drain_grace(defaultDEFAULT_DRAIN_GRACE, 25 s — under the 30 s default lease and Kubernetes' 30 s pod-termination grace). An attempt that outlives the grace is aborted and deliberately not settled: reportingcancelledwould kill the task (terminal, never retried), while leaving it unsettled releases it back to visibility at lease expiry for a worker that is still serving. Wire the token to SIGTERM/SIGINT in your binary — seeexamples/activity_worker_demo.rsfor the idiomatic wiring. - Failure classification uses the shared
ErrorClasstaxonomy:LlmandToolerrors reach/failasdependency_failurewithretryable: true(the transient executor classes);Graph/InvalidUpdatemap toinvalid_input; everything else isunknownwithretryable: false. AnInterrupterror settles as a non-retryablecancelled— the task-queue protocol has no suspend semantics (HITL wiring is the run-outbox wave's concern). - Claim/heartbeat/settle transport failures are retried forever with capped exponential backoff on the claim poll; in a durable system the server coming back is the normal case.
with_leaseclamps to the server's accepted range (100 ms – 1 h).
use Duration;
use ;
use CancellationToken;
# async
API
| Item | Purpose |
|---|---|
WorkerRegistry |
Named Node handlers; new / register / with / contains / len / names / handler. |
router(registry) -> Router |
axum router (POST /execute + GET /ok) for embedding or tests with an ephemeral listener. |
serve(registry, addr) -> io::Result<()> |
Bind and serve until the process stops. |
probe_body() -> Value |
A valid NodeTask JSON body with the current PROTOCOL_VERSION, for manual curl probes. |
ActivityWorker |
Pull-based durable worker: new / register / with_worker_id / with_lease / with_pools / with_claim_backoff / with_drain_grace / worker_id / run(shutdown). |
Activity, ActivityContext |
One durable unit of work (closure-implementable) and its input: task_id / kind / attempt / idempotency_key / payload. |
activity::ClaimedTask |
The claim envelope fields a worker consumes. |
activity::DEFAULT_LEASE and friends |
Defaults for lease, request timeout, claim backoff, and drain grace. |
Demo
Serves a greeter handler and an interrupting approval_gate HITL handler
on 127.0.0.1:8200, and prints the matching RemoteNode wiring and curl
probe commands.
Tests
Unit tests cover the registry (register/replace/builder, probe shape) and the
activity wire shapes (claim/heartbeat/complete/fail bodies, the ErrorClass
classification, the claim-envelope decode); the remote e2e suite runs real
graphs mixing local and remote nodes through the actual Executor — including
an interrupt → resume round trip across the wire — plus the HTTP-layer
contract (protocol-version 400, unknown handler, handler error, and handler
panic all as 200 + one-payload error bodies). The activity e2e suite drives
ActivityWorker against a mock task queue implementing the lease contract
exactly: claim → execute → complete, heartbeats keeping the lease, 409
mid-run aborting the activity, graceful drain finishing in-flight work,
drain-grace expiry aborting the attempt and releasing it for reassignment
(unsettled — never cancelled), drain idempotence, no new claims after
drain starts even with tasks queued, failure classification with the
retryable flag, interrupt-as-cancellation,
panics, unknown kinds, and undecodable claim bodies.