rusty_worker/activity.rs
1//! Durable activities (R0.6): [`ActivityWorker`], the leased-task execution
2//! loop against the rusty-agent-server task queue.
3//!
4//! Where [`crate::serve`] answers one-shot `/execute` calls pushed by a
5//! [`RemoteNode`](rusty_agent_runtime::remote::RemoteNode), an
6//! `ActivityWorker` *pulls* work: it claims tasks from the server under a
7//! lease, renews that lease with heartbeats while the handler runs, and
8//! settles the task with a complete or fail call. The server re-queues tasks
9//! whose lease expires, so a crashed worker never strands work. The promise
10//! is *effectively-once* execution when activity side effects are idempotent
11//! — not universal exactly-once.
12//!
13//! ## The task model
14//!
15//! A task is a generic unit of durable work: a `kind` label plus an
16//! arbitrary JSON `payload`, enqueued via `POST /tasks` and claimed by pool.
17//! The worker registers one [`Activity`] handler per `kind`; the handler
18//! receives an [`ActivityContext`] (task id, attempt, idempotency key,
19//! payload) and returns the JSON result stored on the task record.
20//!
21//! ## The lease protocol (client side)
22//!
23//! - `POST {base}/tasks/claim` — body `{worker_id, pools?, worker_version?,
24//! lease_ms}`; a `200` carries `{"task": {…}}` (a [`ClaimedTask`]), a `204`
25//! means no work is available in the claimed pools. `worker_version` (R0.6
26//! wave 3) is the worker's advertised version: the server matches it
27//! exactly against a task's version pin, so a run pinned to a version
28//! keeps dispatching to workers running it until the run finishes.
29//! - `POST {base}/tasks/{task_id}/heartbeat` — body `{worker_id, lease_ms}`,
30//! sent from a background task every `lease / 3`; a `200` renews the lease
31//! (`{lease_expires_at, cancel_requested}`), a `409` means the lease is
32//! lost.
33//! - `POST {base}/tasks/{task_id}/complete` — body `{worker_id, result}`,
34//! where `result` is the handler's JSON return value; an activity that
35//! performed an idempotent effect additionally carries its `receipt`
36//! (the provider's confirmation — see [`Activity::run_with_receipt`]),
37//! which the server journals into the task's run as an `effect_receipt`
38//! event.
39//! - `POST {base}/tasks/{task_id}/fail` — body
40//! `{worker_id, error_class, message, retryable}`; the reply
41//! `{requeued, next_attempt_at, dead}` is logged for operators.
42//!
43//! A `409` from any of these calls means the server considers the task lost
44//! or already settled; the worker then abandons the activity without further
45//! calls, so the server's reassignment stays the single source of truth.
46//!
47//! ## Semantics
48//!
49//! - **One activity in flight.** The loop claims, executes, and settles one
50//! task at a time. Run several `ActivityWorker` instances for parallelism;
51//! pool routing ([`ActivityWorker::with_pools`]) keeps them on disjoint
52//! queues.
53//! - **Lease loss aborts execution.** When a heartbeat answers `409`, the
54//! handler future is aborted (dropped at its next yield point) and the
55//! worker makes no settle call — the server will reassign the task.
56//! Handler code must tolerate being abandoned mid-effect; that is the
57//! effectively-once contract, and it is why activity side effects should
58//! be keyed by [`ActivityContext::task_id`] /
59//! [`ActivityContext::idempotency_key`].
60//! - **Cancellation aborts promptly, and reports.** When a heartbeat
61//! carries `cancel_requested: true` — or the task's whole-task
62//! `deadline` passes mid-attempt — the handler is aborted the same way,
63//! but the worker *does* settle: it reports the attempt as
64//! [`ErrorClass::Cancelled`] through `/fail`, which the server's shared
65//! retry policy fails outright, so the record ends terminal-cancelled —
66//! never retried, never dead-lettered. A task claimed with the deadline
67//! already passed (or the cancel flag already set) is reported cancelled
68//! without running the handler at all.
69//! - **Graceful drain (R0.6 wave 2c).** Cancelling the shutdown
70//! [`CancellationToken`] *is* the drain request — idempotent, and
71//! race-safe against an in-flight claim poll (the poll itself is
72//! `select!`ed against the token). Draining stops claiming immediately;
73//! an in-flight activity keeps heartbeating and settles normally,
74//! bounded by [`ActivityWorker::with_drain_grace`] (default
75//! [`DEFAULT_DRAIN_GRACE`], 25 s — under the 30 s default lease and
76//! Kubernetes' 30 s pod-termination grace). An attempt that outlives
77//! the grace is aborted and deliberately **not** settled: reporting it
78//! [`ErrorClass::Cancelled`] would kill the task (cancelled is
79//! terminal), while leaving it unsettled releases it back to visibility
80//! at lease expiry for another worker — the design's "finishes or
81//! fast-fails in-flight attempts within a grace period, and releases
82//! the rest".
83//! - **Failure classification.** Handler errors reach `/fail` under the
84//! frozen [`ErrorClass`] taxonomy (shared with the server scheduler via
85//! `rusty_agent_runtime::durable`), with the `retryable` flag mirroring
86//! the executor's judgment: `Llm` and `Tool` failures are the transient
87//! classes; everything else is reported non-retryable.
88//!
89//! ```no_run
90//! use std::time::Duration;
91//! use rusty_worker::{ActivityContext, ActivityWorker};
92//! use tokio_util::sync::CancellationToken;
93//!
94//! # async fn demo() {
95//! let worker = ActivityWorker::new("http://127.0.0.1:8080")
96//! .register("send_receipt", |ctx: ActivityContext| async move {
97//! // `task_id` / `idempotency_key` are the side-effect correlation
98//! // handles that make redelivery effectively-once.
99//! let _dedup_key = ctx.idempotency_key().unwrap_or_else(|| ctx.task_id());
100//! let to = ctx.payload()["to"].as_str().unwrap_or("unknown");
101//! // … perform the effect …
102//! Ok(serde_json::json!({"sent": true, "to": to}))
103//! })
104//! .with_worker_id("email-worker-1")
105//! .with_lease(Duration::from_secs(30))
106//! .with_pools(["email"]);
107//!
108//! // Cancel the token (e.g. from a SIGTERM handler) to drain and exit.
109//! worker.run(CancellationToken::new()).await;
110//! # }
111//! ```
112
113use std::collections::HashMap;
114use std::future::Future;
115use std::sync::Arc;
116use std::time::Duration;
117
118use async_trait::async_trait;
119use chrono::{DateTime, Utc};
120use reqwest::StatusCode;
121use rusty_agent_runtime::durable::ErrorClass;
122use rusty_agent_runtime::error::{Result, RustyError};
123use rusty_agent_runtime::record::EffectReceipt;
124use serde::{Deserialize, Serialize};
125use serde_json::Value;
126use tokio_util::sync::CancellationToken;
127use tracing::Instrument;
128use uuid::Uuid;
129
130/// Default lease requested on claim and renewed by heartbeats (30 s).
131pub const DEFAULT_LEASE: Duration = Duration::from_secs(30);
132
133/// Default per-request HTTP timeout for claim/heartbeat/settle calls (10 s).
134pub const DEFAULT_REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
135
136/// Default base delay between claim polls when no work is available (or the
137/// server is unreachable); consecutive empty polls back off exponentially up
138/// to [`DEFAULT_CLAIM_BACKOFF_MAX`].
139pub const DEFAULT_CLAIM_BACKOFF_BASE: Duration = Duration::from_millis(100);
140
141/// Default cap for the claim-poll backoff.
142pub const DEFAULT_CLAIM_BACKOFF_MAX: Duration = Duration::from_secs(5);
143
144/// Default drain grace (25 s): once draining starts, an in-flight activity
145/// gets this long to finish and settle before the worker aborts it and
146/// exits, leaving the attempt for the server to reassign at lease expiry.
147///
148/// Chosen to sit under the [`DEFAULT_LEASE`] of 30 s — the drain never
149/// outlives the visibility window the lease already provides — and under
150/// Kubernetes' default 30 s pod-termination grace, so a drained worker
151/// exits before SIGKILL would arrive. Override with
152/// [`ActivityWorker::with_drain_grace`]; when you also change the lease,
153/// keep the grace below it.
154pub const DEFAULT_DRAIN_GRACE: Duration = Duration::from_secs(25);
155
156// The server's accepted `lease_ms` range (`tasks::MIN_LEASE_MS..=MAX_LEASE_MS`
157// in rusty-agent-server): below 100 ms a lease is an instant expiry, above one
158// hour a lost worker strands its task for too long. `with_lease` clamps to
159// this range so a misconfigured worker gets a sane lease instead of a 400
160// on every claim.
161const MIN_LEASE: Duration = Duration::from_millis(100);
162const MAX_LEASE: Duration = Duration::from_secs(3_600);
163
164/// The input to every activity invocation: the claimed task's identity,
165/// attempt ordinal, dedup key, and payload.
166///
167/// `task_id` (and, when the enqueue side set one, `idempotency_key`) are
168/// the correlation handles for external side effects — a redelivered task
169/// (lease expiry, crash) re-runs the handler with the same ids, which is
170/// what makes the effect effectively-once.
171#[derive(Debug, Clone)]
172pub struct ActivityContext {
173 task_id: String,
174 kind: String,
175 attempt: u32,
176 idempotency_key: Option<String>,
177 payload: Value,
178}
179
180impl ActivityContext {
181 /// The server-minted task id.
182 pub fn task_id(&self) -> &str {
183 &self.task_id
184 }
185
186 /// The task kind this invocation was dispatched under.
187 pub fn kind(&self) -> &str {
188 &self.kind
189 }
190
191 /// The 1-based attempt ordinal (`1` on first delivery, higher after a
192 /// requeue).
193 pub fn attempt(&self) -> u32 {
194 self.attempt
195 }
196
197 /// The dedup key supplied at enqueue time, if any.
198 pub fn idempotency_key(&self) -> Option<&str> {
199 self.idempotency_key.as_deref()
200 }
201
202 /// The task payload, exactly as enqueued.
203 pub fn payload(&self) -> &Value {
204 &self.payload
205 }
206}
207
208/// An activity: one durable unit of work, dispatched by task `kind`.
209///
210/// Implement this directly for stateful activities, or just pass an async
211/// closure `Fn(ActivityContext) -> impl Future<Output = Result<Value>>` to
212/// [`ActivityWorker::register`] — a blanket impl covers it, mirroring the
213/// [`Node`](rusty_agent_runtime::node::Node) ergonomics.
214///
215/// The returned `Value` is stored verbatim as the task's `result`. An
216/// `Err` is classified into the shared [`ErrorClass`] taxonomy and reported
217/// to `POST /tasks/{id}/fail`.
218///
219/// Activities performing an `Idempotent` effect should override
220/// [`Activity::run_with_receipt`] instead: the receipt it returns — the
221/// provider's own confirmation, under the task's idempotency key — travels
222/// with the completion and is journaled into the task's run as evidence
223/// exact replay can serve instead of re-sending the effect.
224#[async_trait]
225pub trait Activity: Send + Sync {
226 /// Execute the activity against a claimed task.
227 async fn run(&self, ctx: ActivityContext) -> Result<Value>;
228
229 /// Execute the activity, additionally reporting an effect receipt with
230 /// the result. The default wraps [`Activity::run`] with no receipt, so
231 /// existing activities (and the closure blanket impl) are unaffected —
232 /// a receipt is something an activity opts into reporting, never
233 /// something the worker invents on its behalf: only the code that
234 /// performed the effect knows the provider's confirmation id.
235 async fn run_with_receipt(&self, ctx: ActivityContext) -> Result<ActivityCompletion> {
236 Ok(ActivityCompletion {
237 result: self.run(ctx).await?,
238 receipt: None,
239 })
240 }
241}
242
243/// What an activity reports on success: the stored `result` plus, when the
244/// activity performed an `Idempotent` effect, its [`EffectReceipt`] — the
245/// effect's own confirmation (provider id, idempotency key), which the
246/// server journals into the task's run as a `effect_receipt` event.
247#[derive(Debug, Clone)]
248pub struct ActivityCompletion {
249 /// The value stored as the task's `result` (any JSON).
250 pub result: Value,
251
252 /// The effect's confirmation, when the activity performed an idempotent
253 /// effect and the provider confirmed it. `None` for pure/read-only work
254 /// and for effects that carry no confirmation handle.
255 pub receipt: Option<EffectReceipt>,
256}
257
258/// Blanket implementation for async closures/functions:
259/// `Fn(ActivityContext) -> impl Future<Output = Result<Value>>`.
260#[async_trait]
261impl<F, Fut> Activity for F
262where
263 F: Fn(ActivityContext) -> Fut + Send + Sync,
264 Fut: Future<Output = Result<Value>> + Send,
265{
266 async fn run(&self, ctx: ActivityContext) -> Result<Value> {
267 (self)(ctx).await
268 }
269}
270
271/// Allow `Arc<dyn Activity>` itself to be registered (useful for sharing one
272/// activity implementation across workers).
273///
274/// Both methods delegate: overriding only `run` would make the default
275/// `run_with_receipt` answer for the `Arc` — wrapping the inner `run` and
276/// silently dropping the receipt a receipt-carrying activity reported. The
277/// worker stores every registered handler as `Arc<dyn Activity>` and drives
278/// it through `run_with_receipt`, so this delegation is what carries the
279/// effect receipt from the handler to the completion call.
280#[async_trait]
281impl Activity for Arc<dyn Activity> {
282 async fn run(&self, ctx: ActivityContext) -> Result<Value> {
283 self.as_ref().run(ctx).await
284 }
285
286 async fn run_with_receipt(&self, ctx: ActivityContext) -> Result<ActivityCompletion> {
287 self.as_ref().run_with_receipt(ctx).await
288 }
289}
290
291/// The task a worker receives on a successful claim: the fields of the
292/// server's task record a worker needs (`POST /tasks/claim` →
293/// `{"task": {…}}`). Fields the worker does not consume are ignored on
294/// deserialization, so additive server-side evolution keeps working.
295#[derive(Debug, Clone, Deserialize)]
296pub struct ClaimedTask {
297 /// The server-minted task id, used to address heartbeat/complete/fail.
298 pub task_id: String,
299
300 /// The dispatch label the worker's handler registry is keyed by.
301 pub kind: String,
302
303 /// The work payload, exactly as enqueued.
304 #[serde(default)]
305 pub payload: Value,
306
307 /// The 1-based attempt ordinal (higher after a requeue).
308 #[serde(default)]
309 pub attempt: u32,
310
311 /// The attempt ceiling configured at enqueue time.
312 #[serde(default)]
313 pub max_attempts: u32,
314
315 /// The dedup key supplied at enqueue time, if any.
316 #[serde(default)]
317 pub idempotency_key: Option<String>,
318
319 /// Run linkage (reserved for the run-side outbox wave); carried into the
320 /// activity's tracing span when present.
321 #[serde(default)]
322 pub run_id: Option<String>,
323
324 /// See [`ClaimedTask::run_id`].
325 #[serde(default)]
326 pub thread_id: Option<String>,
327
328 /// Cancellation signalled before the claim was handed out. A
329 /// well-behaved server never leases such a task (its claim path
330 /// finalizes them as cancelled), but the worker defends anyway: a
331 /// claimed task with this flag is reported cancelled without running
332 /// the handler.
333 #[serde(default)]
334 pub cancel_requested: bool,
335
336 /// Whole-task deadline, across attempts. The worker refuses to start
337 /// an attempt past it and aborts a running attempt at it, reporting
338 /// [`ErrorClass::Cancelled`] — deadline expiry is cancellation by
339 /// clock, per the Durable Work design.
340 #[serde(default)]
341 pub deadline: Option<DateTime<Utc>>,
342}
343
344/// `POST /tasks/claim` request body.
345#[derive(Debug, Serialize)]
346struct ClaimRequest<'a> {
347 worker_id: &'a str,
348 #[serde(skip_serializing_if = "Option::is_none")]
349 pools: Option<&'a [String]>,
350 /// The worker's advertised version (R0.6 wave 3), matched exactly
351 /// against a task's `worker_version` pin. Omitted (not null) when
352 /// unconfigured — an unversioned worker takes unpinned work only, and
353 /// pre-wave-3 servers never see the field.
354 #[serde(skip_serializing_if = "Option::is_none")]
355 worker_version: Option<&'a str>,
356 lease_ms: u64,
357}
358
359/// `POST /tasks/claim` success body: the task, wrapped.
360#[derive(Debug, Deserialize)]
361struct ClaimResponse {
362 task: ClaimedTask,
363}
364
365/// `POST /tasks/{id}/heartbeat` request body.
366#[derive(Debug, Serialize)]
367struct HeartbeatRequest<'a> {
368 worker_id: &'a str,
369 lease_ms: u64,
370}
371
372/// `POST /tasks/{id}/heartbeat` success body. `lease_expires_at` is an
373/// RFC 3339 timestamp; the worker treats it as informational (the server's
374/// `409` is the authoritative lease-loss signal) and logs it.
375/// `cancel_requested` is the cancellation hint: on `true` the worker
376/// aborts the in-flight attempt and reports it [`ErrorClass::Cancelled`].
377/// Absent on pre-wave-2 servers, hence the default.
378#[derive(Debug, Deserialize)]
379struct HeartbeatResponse {
380 lease_expires_at: Option<String>,
381 #[serde(default)]
382 cancel_requested: bool,
383}
384
385/// `POST /tasks/{id}/complete` request body. `receipt` is omitted (not
386/// null) when the activity reports none, keeping the body identical for
387/// pre-receipt activities.
388#[derive(Debug, Serialize)]
389struct CompleteRequest<'a> {
390 worker_id: &'a str,
391 result: &'a Value,
392 #[serde(skip_serializing_if = "Option::is_none")]
393 receipt: Option<&'a EffectReceipt>,
394}
395
396/// `POST /tasks/{id}/fail` request body. `error_class` serializes to the
397/// frozen snake_case wire spelling of [`ErrorClass`].
398#[derive(Debug, Serialize)]
399struct FailRequest<'a> {
400 worker_id: &'a str,
401 error_class: ErrorClass,
402 message: &'a str,
403 retryable: bool,
404}
405
406/// `POST /tasks/{id}/fail` success body: the server's retry decision.
407#[derive(Debug, Deserialize)]
408struct FailResponse {
409 requeued: bool,
410 next_attempt_at: Option<String>,
411 dead: bool,
412}
413
414/// Classify a handler error for the `/fail` call.
415///
416/// Two independent judgments, both owned by the executor of the work:
417///
418/// - `error_class` — the frozen [`ErrorClass`] taxonomy shared with the
419/// server scheduler. `Llm` and `Tool` failures are upstream
420/// ([`ErrorClass::DependencyFailure`]); `Graph` and `InvalidUpdate` are
421/// contract violations ([`ErrorClass::InvalidInput`]); everything
422/// unclassifiable is [`ErrorClass::Unknown`]. An `Interrupt` error is
423/// [`ErrorClass::Cancelled`]: the task-queue protocol has no suspend
424/// semantics (HITL wiring is the run-outbox wave's concern), so a handler
425/// interrupt settles as a non-retryable cancellation.
426/// - `retryable` — the executor's own taxonomy: `Llm` and `Tool` are the
427/// transient classes; everything else is a hard failure.
428fn classify_error(error: &RustyError) -> (ErrorClass, bool) {
429 match error {
430 RustyError::Llm(_) => (ErrorClass::DependencyFailure, true),
431 RustyError::Tool(_) => (ErrorClass::DependencyFailure, true),
432 RustyError::Node(_) => (ErrorClass::Unknown, false),
433 RustyError::Graph(_) => (ErrorClass::InvalidInput, false),
434 RustyError::InvalidUpdate(_) => (ErrorClass::InvalidInput, false),
435 RustyError::Checkpoint(_) => (ErrorClass::Unknown, false),
436 RustyError::Replay(_) => (ErrorClass::Unknown, false),
437 RustyError::Serialization(_) => (ErrorClass::Unknown, false),
438 RustyError::Interrupt { .. } => (ErrorClass::Cancelled, false),
439 // A handler whose own inner run was drained propagates the
440 // cancellation class: control flow, never retried.
441 RustyError::Cancelled(_) => (ErrorClass::Cancelled, false),
442 }
443}
444
445/// The outcome of one claim poll.
446enum ClaimOutcome {
447 /// A task was claimed and must be executed and settled. Boxed: the
448 /// claimed-task payload dwarfs the other variants and the wasm-enabled
449 /// feature combination pushes it past clippy's size threshold.
450 Task(Box<ClaimedTask>),
451 /// The server answered `204`: no work available.
452 Empty,
453 /// The poll failed (transport error, unexpected status, undecodable
454 /// body); the caller backs off and polls again.
455 Unavailable,
456}
457
458/// Why the attempt runner stopped waiting on the handler future: finished
459/// normally, the lease is gone (abandon — the server reassigns), or the
460/// task was cancelled (abort, then report [`ErrorClass::Cancelled`]
461/// through the fail path while we still hold the lease), or the drain
462/// grace expired (abort, then leave the attempt unsettled — see
463/// [`ActivityWorker::run`]'s drain semantics).
464enum AttemptOutcome {
465 /// The handler future completed (possibly with an error or panic).
466 Finished(std::result::Result<Result<ActivityCompletion>, tokio::task::JoinError>),
467 /// A heartbeat answered `409`: the server considers the task lost or
468 /// settled. The worker makes no further calls for it.
469 LeaseLost,
470 /// Cancellation reached the worker — a heartbeat carried
471 /// `cancel_requested: true`, or the whole-task deadline expired. The
472 /// payload is the reason recorded on the fail report.
473 Cancelled(&'static str),
474 /// The worker is draining and the in-flight attempt outlived the
475 /// configured grace. Like [`AttemptOutcome::LeaseLost`] the attempt is
476 /// abandoned without a settle call — but deliberately, not because the
477 /// lease is gone: settling `cancelled` would kill the task outright,
478 /// while leaving it unsettled lets lease expiry hand it to a worker
479 /// that is not being deployed away.
480 DrainGraceExpired,
481}
482
483/// The deadline arm of the attempt runner's `select!`: fires when the
484/// whole-task deadline passes, or pends forever when the task has none —
485/// one arm shape either way.
486async fn sleep_until_deadline(until: Option<Duration>) {
487 match until {
488 Some(delay) => tokio::time::sleep(delay).await,
489 None => std::future::pending().await,
490 }
491}
492
493/// A durable worker that claims leased tasks from the rusty-agent-server task
494/// queue and executes them behind heartbeats.
495///
496/// This is the pull-based counterpart to [`crate::serve`]: instead of
497/// answering one-shot `/execute` calls, it polls `POST /tasks/claim`, runs
498/// the [`Activity`] registered for the claimed task's `kind`, renews the
499/// lease via a background heartbeat every `lease / 3`, and settles with
500/// `complete` or `fail` (classified error). See the
501/// [module documentation](self) for the full protocol and semantics.
502///
503/// Run several instances for parallelism; pool routing keeps them on
504/// disjoint queues.
505pub struct ActivityWorker {
506 handlers: HashMap<String, Arc<dyn Activity>>,
507 /// Server base URL, trailing slashes trimmed.
508 base_url: String,
509 /// The identity sent on every claim/heartbeat/settle call.
510 worker_id: String,
511 client: reqwest::Client,
512 /// Requested lease duration; heartbeats fire at `lease / 3`.
513 lease: Duration,
514 /// Pool names to claim from; empty means the server's default pool.
515 pools: Vec<String>,
516 /// The version advertised on every claim (R0.6 wave 3), matched
517 /// exactly against tasks' `worker_version` pins; `None` claims
518 /// unpinned work only.
519 worker_version: Option<String>,
520 claim_backoff_base: Duration,
521 claim_backoff_max: Duration,
522 /// How long an in-flight activity may run after draining starts before
523 /// the worker aborts it and exits (see [`DEFAULT_DRAIN_GRACE`]).
524 drain_grace: Duration,
525}
526
527impl ActivityWorker {
528 /// A worker claiming tasks from the rusty-agent-server at `base_url` (e.g.
529 /// `"http://127.0.0.1:8080"`).
530 ///
531 /// Register at least one [`Activity`] with [`ActivityWorker::register`]
532 /// before running. The worker id defaults to `worker-{uuid}`; set a
533 /// stable one with [`ActivityWorker::with_worker_id`] if operators
534 /// should recognize the worker across restarts.
535 pub fn new(base_url: impl Into<String>) -> Self {
536 Self {
537 handlers: HashMap::new(),
538 base_url: base_url.into().trim_end_matches('/').to_owned(),
539 worker_id: format!("worker-{}", Uuid::new_v4()),
540 // Same construction invariant as `RemoteNode`: the builder only
541 // sets a timeout and the rustls backend needs no platform
542 // initialization, so this cannot realistically fail.
543 client: reqwest::Client::builder()
544 .timeout(DEFAULT_REQUEST_TIMEOUT)
545 .build()
546 .expect("reqwest client builder with rustls must succeed"),
547 lease: DEFAULT_LEASE,
548 pools: Vec::new(),
549 worker_version: None,
550 claim_backoff_base: DEFAULT_CLAIM_BACKOFF_BASE,
551 claim_backoff_max: DEFAULT_CLAIM_BACKOFF_MAX,
552 drain_grace: DEFAULT_DRAIN_GRACE,
553 }
554 }
555
556 /// Register the activity executed for tasks of `kind`.
557 ///
558 /// Accepts any [`Activity`] implementation — including plain async
559 /// closures via the blanket impl. Registering the same kind twice
560 /// replaces the previous handler.
561 pub fn register<A>(mut self, kind: impl Into<String>, activity: A) -> Self
562 where
563 A: Activity + 'static,
564 {
565 self.handlers.insert(kind.into(), Arc::new(activity));
566 self
567 }
568
569 /// Override the worker identity sent on every call to the server.
570 pub fn with_worker_id(mut self, worker_id: impl Into<String>) -> Self {
571 self.worker_id = worker_id.into();
572 self
573 }
574
575 /// Override the requested lease duration. Heartbeats renew the lease
576 /// from a background task every `lease / 3` (clamped to ≥ 1 ms), so a
577 /// lease must comfortably outlive three heartbeat intervals of clock
578 /// drift and scheduling jitter — the [`DEFAULT_LEASE`] of 30 s is a good
579 /// floor for real deployments. The value is clamped to the server's
580 /// accepted range (100 ms – 1 h).
581 pub fn with_lease(mut self, lease: Duration) -> Self {
582 self.lease = lease.clamp(MIN_LEASE, MAX_LEASE);
583 self
584 }
585
586 /// Restrict claiming to the named pools (`"pools": [...]` on the claim
587 /// call). With no pools configured the claim omits the field and the
588 /// server's default pool applies.
589 pub fn with_pools<I, S>(mut self, pools: I) -> Self
590 where
591 I: IntoIterator<Item = S>,
592 S: Into<String>,
593 {
594 self.pools = pools.into_iter().map(Into::into).collect();
595 self
596 }
597
598 /// Advertise this worker's version on every claim (R0.6 wave 3). The
599 /// server matches it **exactly** against a task's `worker_version`
600 /// pin: tasks a run pinned to this version are leased to this worker
601 /// (and its version-mates) until the run finishes, so a mid-run deploy
602 /// never changes semantics under an in-flight execution; unpinned
603 /// tasks remain claimable regardless. With no version configured the
604 /// claim omits the field and the worker takes unpinned work only —
605 /// a worker that should serve pinned runs must advertise the version
606 /// it actually runs. Free-form string, but make it the real build
607 /// identity (crate version, image tag): a stale advertisement defeats
608 /// the pin's purpose.
609 pub fn with_worker_version(mut self, version: impl Into<String>) -> Self {
610 self.worker_version = Some(version.into());
611 self
612 }
613
614 /// Override the claim-poll backoff: empty or failed polls wait roughly
615 /// `base * 2^n` (capped at `max`) until a task is claimed.
616 pub fn with_claim_backoff(mut self, base: Duration, max: Duration) -> Self {
617 self.claim_backoff_base = base;
618 self.claim_backoff_max = max;
619 self
620 }
621
622 /// Override the drain grace (default [`DEFAULT_DRAIN_GRACE`]): how long
623 /// an in-flight activity may keep running after the shutdown token
624 /// fires before the worker aborts it and exits. An aborted attempt is
625 /// deliberately **not** settled — it returns to visibility at lease
626 /// expiry for another worker — so keep the grace under the lease:
627 /// beyond the lease the abort buys nothing (the server would reassign
628 /// at the same moment either way).
629 pub fn with_drain_grace(mut self, grace: Duration) -> Self {
630 self.drain_grace = grace;
631 self
632 }
633
634 /// The identity this worker presents to the server.
635 pub fn worker_id(&self) -> &str {
636 &self.worker_id
637 }
638
639 /// Run the claim → execute → settle loop until `shutdown` is cancelled,
640 /// then drain: stop claiming, settle any in-flight activity, and return.
641 ///
642 /// Cancelling `shutdown` **is** the drain request — there is no separate
643 /// `drain()` because the token already is the control: idempotent
644 /// (cancelling twice changes nothing), race-safe (a cancel racing an
645 /// in-flight claim poll still wins, because the claim itself is
646 /// `select!`ed against the token), and shareable (every clone of the
647 /// token observes the same signal). Wire it to SIGTERM/SIGINT in the
648 /// embedding binary; see the crate's `activity_worker_demo` example.
649 ///
650 /// Drain semantics: claiming stops immediately; an in-flight activity
651 /// keeps heartbeating and settles normally, bounded by
652 /// [`ActivityWorker::with_drain_grace`]. An attempt still running when
653 /// the grace expires is aborted and deliberately **not** settled —
654 /// reporting it [`ErrorClass::Cancelled`] would kill the task
655 /// (cancelled is terminal: never retried, never dead-lettered), which
656 /// is exactly what a deployment must not do. Left unsettled, the
657 /// attempt returns to visibility at lease expiry and another worker
658 /// picks it up — the design's "releases the rest, which return to
659 /// visibility for other workers".
660 ///
661 /// The loop never gives up on the server: claim/heartbeat/settle
662 /// transport failures are logged and retried with backoff, because in a
663 /// durable system the server coming back is the normal case.
664 pub async fn run(self, shutdown: CancellationToken) {
665 tracing::info!(
666 worker_id = %self.worker_id,
667 base_url = %self.base_url,
668 lease_ms = self.lease_ms(),
669 pools = ?self.pools,
670 worker_version = %self.worker_version.as_deref().unwrap_or(""),
671 kinds = ?self.registered_kinds(),
672 drain_grace_ms = self.drain_grace.as_millis() as u64,
673 "activity worker started"
674 );
675 let mut backoff = self.claim_backoff_base;
676 loop {
677 // The claim poll itself is raced against the token: a drain
678 // that starts mid-poll still stops this worker from taking the
679 // lease. (If the server answered just as the poll was dropped,
680 // the task is leased to a worker that never runs it — it
681 // returns at lease expiry, the same backstop as a crash.)
682 let claim = tokio::select! {
683 _ = shutdown.cancelled() => break,
684 outcome = self.claim() => outcome,
685 };
686 match claim {
687 ClaimOutcome::Task(task) => {
688 backoff = self.claim_backoff_base;
689 self.run_activity(*task, &shutdown).await;
690 }
691 ClaimOutcome::Empty | ClaimOutcome::Unavailable => {
692 let delay = backoff;
693 backoff = backoff
694 .checked_mul(2)
695 .unwrap_or(self.claim_backoff_max)
696 .min(self.claim_backoff_max);
697 tokio::select! {
698 _ = tokio::time::sleep(delay) => {}
699 _ = shutdown.cancelled() => {}
700 }
701 }
702 }
703 }
704 tracing::info!(worker_id = %self.worker_id, "activity worker drained and stopped");
705 }
706
707 fn registered_kinds(&self) -> Vec<&str> {
708 let mut kinds: Vec<&str> = self.handlers.keys().map(String::as_str).collect();
709 kinds.sort_unstable();
710 kinds
711 }
712
713 fn lease_ms(&self) -> u64 {
714 u64::try_from(self.lease.as_millis()).unwrap_or(u64::MAX)
715 }
716
717 /// One claim poll against `POST /tasks/claim`.
718 async fn claim(&self) -> ClaimOutcome {
719 let body = ClaimRequest {
720 worker_id: &self.worker_id,
721 pools: if self.pools.is_empty() {
722 None
723 } else {
724 Some(self.pools.as_slice())
725 },
726 worker_version: self.worker_version.as_deref(),
727 lease_ms: self.lease_ms(),
728 };
729 let response = match self
730 .client
731 .post(format!("{}/tasks/claim", self.base_url))
732 .json(&body)
733 .send()
734 .await
735 {
736 Ok(response) => response,
737 Err(e) => {
738 tracing::warn!(error = %e, "claim request failed");
739 return ClaimOutcome::Unavailable;
740 }
741 };
742 match response.status() {
743 StatusCode::OK => match response.json::<ClaimResponse>().await {
744 Ok(claimed) => ClaimOutcome::Task(Box::new(claimed.task)),
745 Err(e) => {
746 tracing::warn!(error = %e, "claim returned an undecodable task body");
747 ClaimOutcome::Unavailable
748 }
749 },
750 StatusCode::NO_CONTENT => ClaimOutcome::Empty,
751 status => {
752 tracing::warn!(%status, "claim rejected by server");
753 ClaimOutcome::Unavailable
754 }
755 }
756 }
757
758 /// Execute one claimed task and settle it, inside a `rusty.activity`
759 /// span. The one case without a settle call is lease loss: the server
760 /// owns the task again, so any call from us would `409`. (Drain-grace
761 /// expiry is the deliberate second case — see
762 /// [`AttemptOutcome::DrainGraceExpired`].)
763 async fn run_activity(&self, task: ClaimedTask, shutdown: &CancellationToken) {
764 let span = tracing::info_span!(
765 "rusty.activity",
766 worker_id = %self.worker_id,
767 task_id = %task.task_id,
768 kind = %task.kind,
769 attempt = task.attempt,
770 max_attempts = task.max_attempts,
771 run_id = %task.run_id.as_deref().unwrap_or(""),
772 thread_id = %task.thread_id.as_deref().unwrap_or(""),
773 );
774 async move {
775 let Some(handler) = self.handlers.get(&task.kind).cloned() else {
776 self.fail(
777 &task.task_id,
778 ErrorClass::InvalidInput,
779 format!(
780 "no activity registered for kind `{}` on this worker (registered: {:?})",
781 task.kind,
782 self.registered_kinds()
783 ),
784 false,
785 )
786 .await;
787 return;
788 };
789
790 // Work that is already cancelled must not start: report the
791 // attempt cancelled without running the handler. (A
792 // well-behaved server never leases such a task; this is the
793 // worker-side defense of the same rule.)
794 if task.cancel_requested {
795 tracing::warn!("claimed task was already cancel-requested; reporting cancelled");
796 self.fail(
797 &task.task_id,
798 ErrorClass::Cancelled,
799 format!("activity `{}` cancelled before dispatch", task.kind),
800 false,
801 )
802 .await;
803 return;
804 }
805 // Wall-clock remaining until the whole-task deadline; a
806 // deadline already passed is reported cancelled without
807 // running the handler.
808 let until_deadline = match task.deadline {
809 Some(deadline) => match (deadline - Utc::now()).to_std() {
810 Ok(remaining) => Some(remaining),
811 Err(_) => {
812 tracing::warn!(%deadline, "claimed task's deadline already passed; reporting cancelled");
813 self.fail(
814 &task.task_id,
815 ErrorClass::Cancelled,
816 format!(
817 "activity `{}` cancelled: whole-task deadline already passed",
818 task.kind
819 ),
820 false,
821 )
822 .await;
823 return;
824 }
825 },
826 None => None,
827 };
828
829 let ctx = ActivityContext {
830 task_id: task.task_id.clone(),
831 kind: task.kind.clone(),
832 attempt: task.attempt,
833 idempotency_key: task.idempotency_key.clone(),
834 payload: task.payload.clone(),
835 };
836 let task_id = task.task_id.clone();
837
838 let lease_lost = CancellationToken::new();
839 let cancel_requested = CancellationToken::new();
840 let heartbeat =
841 self.spawn_heartbeat(&task_id, lease_lost.clone(), cancel_requested.clone());
842
843 // The handler runs on its own task for the same reason as in
844 // `/execute`: a panic must surface as an outcome, not a
845 // torn-down worker.
846 let mut handle = tokio::spawn(async move { handler.run_with_receipt(ctx).await });
847 // The drain bound: starts counting when the drain starts (not
848 // at claim time), so a worker idle for hours and drained
849 // mid-attempt still gets the full grace for that attempt.
850 let drain_grace = self.drain_grace;
851 let drained = shutdown.clone();
852 let outcome = tokio::select! {
853 joined = &mut handle => AttemptOutcome::Finished(joined),
854 _ = lease_lost.cancelled() => AttemptOutcome::LeaseLost,
855 _ = cancel_requested.cancelled() => AttemptOutcome::Cancelled("cancelled by the control plane"),
856 _ = sleep_until_deadline(until_deadline) => AttemptOutcome::Cancelled("whole-task deadline expired mid-attempt"),
857 _ = async {
858 drained.cancelled().await;
859 tokio::time::sleep(drain_grace).await;
860 } => AttemptOutcome::DrainGraceExpired,
861 };
862 heartbeat.abort();
863
864 let joined = match outcome {
865 AttemptOutcome::Finished(joined) => joined,
866 AttemptOutcome::LeaseLost => {
867 // Dropping the JoinHandle alone would detach the handler and
868 // let it keep running; abort it and await the abort so the
869 // handler future is really dropped before the next claim.
870 handle.abort();
871 let _ = handle.await;
872 tracing::warn!("lease lost; activity aborted and left for the server to reassign");
873 return;
874 }
875 AttemptOutcome::DrainGraceExpired => {
876 // Same abort discipline as lease loss, and the same "no
877 // settle call" — but by choice: the lease is still
878 // held, and reporting `cancelled` would kill the task
879 // (terminal, never retried). Leaving it unsettled
880 // releases the task back to visibility at lease expiry
881 // for a worker that is still serving.
882 handle.abort();
883 let _ = handle.await;
884 tracing::warn!(
885 drain_grace_ms = drain_grace.as_millis() as u64,
886 "drain grace expired; activity aborted and left for reassignment at lease expiry"
887 );
888 return;
889 }
890 AttemptOutcome::Cancelled(reason) => {
891 // Same abort discipline as lease loss, but here we still
892 // hold the lease: settle the attempt as cancelled so the
893 // record ends terminal-cancelled rather than waiting out
894 // the lease.
895 handle.abort();
896 let _ = handle.await;
897 tracing::warn!(reason, "activity aborted");
898 self.fail(
899 &task_id,
900 ErrorClass::Cancelled,
901 format!("activity `{}` {reason}", task.kind),
902 false,
903 )
904 .await;
905 return;
906 }
907 };
908
909 match joined {
910 Ok(Ok(completion)) => {
911 tracing::info!(receipt = completion.receipt.is_some(), "activity succeeded");
912 self.complete(&task_id, &completion.result, completion.receipt.as_ref())
913 .await;
914 }
915 Ok(Err(e)) => {
916 let (error_class, retryable) = classify_error(&e);
917 tracing::warn!(error = %e, ?error_class, retryable, "activity failed");
918 self.fail(
919 &task_id,
920 error_class,
921 format!("activity `{}` failed: {e}", task.kind),
922 retryable,
923 )
924 .await;
925 }
926 Err(join_err) => {
927 let detail = if join_err.is_panic() {
928 let payload = join_err.into_panic();
929 let message = payload
930 .downcast_ref::<&str>()
931 .map(|s| (*s).to_owned())
932 .or_else(|| payload.downcast_ref::<String>().cloned())
933 .unwrap_or_else(|| "non-string panic payload".to_owned());
934 format!("handler panicked: {message}")
935 } else {
936 "handler task cancelled".to_owned()
937 };
938 tracing::warn!(%detail, "activity panicked");
939 self.fail(
940 &task_id,
941 ErrorClass::Unknown,
942 format!("activity `{}` {detail}", task.kind),
943 false,
944 )
945 .await;
946 }
947 }
948 }
949 .instrument(span)
950 .await
951 }
952
953 /// Spawn the background heartbeat loop for one claimed task: renew the
954 /// lease every `lease / 3`. The loop ends the worker's wait on the
955 /// handler in two cases: a `409` cancels `lease_lost` (the server
956 /// reassigns; the worker abandons), and a `200` carrying
957 /// `cancel_requested: true` cancels `cancel_requested` (the worker
958 /// aborts promptly and reports the attempt as cancelled through the
959 /// fail path — the whole point of the hint).
960 fn spawn_heartbeat(
961 &self,
962 task_id: &str,
963 lease_lost: CancellationToken,
964 cancel_requested: CancellationToken,
965 ) -> tokio::task::JoinHandle<()> {
966 let client = self.client.clone();
967 let url = format!("{}/tasks/{task_id}/heartbeat", self.base_url);
968 let worker_id = self.worker_id.clone();
969 let lease_ms = self.lease_ms();
970 let interval = (self.lease / 3).max(Duration::from_millis(1));
971 tokio::spawn(async move {
972 let mut ticker = tokio::time::interval(interval);
973 // A delayed tick must not burst: heartbeat spacing is a liveness
974 // signal, not a quota to catch up on.
975 ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
976 loop {
977 ticker.tick().await;
978 let body = HeartbeatRequest {
979 worker_id: &worker_id,
980 lease_ms,
981 };
982 match client.post(&url).json(&body).send().await {
983 Ok(response) if response.status() == StatusCode::OK => {
984 match response.json::<HeartbeatResponse>().await {
985 Ok(renewed) => {
986 if renewed.cancel_requested {
987 tracing::warn!(
988 "heartbeat carried cancel_requested; aborting the attempt"
989 );
990 cancel_requested.cancel();
991 return;
992 }
993 let expires_at = renewed
994 .lease_expires_at
995 .unwrap_or_else(|| "<missing>".to_owned());
996 tracing::debug!(lease_expires_at = %expires_at, "lease renewed");
997 }
998 Err(_) => {
999 // An undecodable renewal is not a lease-loss
1000 // signal; the 409 is. Keep heartbeating.
1001 tracing::warn!("heartbeat reply undecodable; will retry");
1002 }
1003 }
1004 }
1005 Ok(response) if response.status() == StatusCode::CONFLICT => {
1006 tracing::warn!("heartbeat rejected (409): lease lost");
1007 lease_lost.cancel();
1008 return;
1009 }
1010 Ok(response) => {
1011 // Any other status is treated as transient: the
1012 // server will answer 409 once the lease is really
1013 // gone, which is the authoritative signal.
1014 tracing::warn!(status = %response.status(), "heartbeat failed; will retry");
1015 }
1016 Err(e) => {
1017 tracing::warn!(error = %e, "heartbeat request failed; will retry");
1018 }
1019 }
1020 }
1021 })
1022 }
1023
1024 /// Settle a task successfully via `POST /tasks/{id}/complete`, carrying
1025 /// the effect receipt when the activity reported one.
1026 async fn complete(&self, task_id: &str, result: &Value, receipt: Option<&EffectReceipt>) {
1027 let body = CompleteRequest {
1028 worker_id: &self.worker_id,
1029 result,
1030 receipt,
1031 };
1032 let response = self
1033 .client
1034 .post(format!("{}/tasks/{task_id}/complete", self.base_url))
1035 .json(&body)
1036 .send()
1037 .await;
1038 self.log_settle("complete", response).await;
1039 }
1040
1041 /// Settle a task as failed via `POST /tasks/{id}/fail` and log the
1042 /// server's retry decision (`{requeued, next_attempt_at, dead}`).
1043 async fn fail(&self, task_id: &str, error_class: ErrorClass, message: String, retryable: bool) {
1044 let body = FailRequest {
1045 worker_id: &self.worker_id,
1046 error_class,
1047 message: &message,
1048 retryable,
1049 };
1050 let response = self
1051 .client
1052 .post(format!("{}/tasks/{task_id}/fail", self.base_url))
1053 .json(&body)
1054 .send()
1055 .await;
1056 match response {
1057 Ok(response) if response.status() == StatusCode::OK => {
1058 match response.json::<FailResponse>().await {
1059 Ok(outcome) => tracing::info!(
1060 requeued = outcome.requeued,
1061 next_attempt_at = ?outcome.next_attempt_at,
1062 dead = outcome.dead,
1063 "task settled as failed"
1064 ),
1065 Err(e) => {
1066 tracing::warn!(error = %e, "task settled as failed; reply undecodable")
1067 }
1068 }
1069 }
1070 other => self.log_settle("fail", other).await,
1071 }
1072 }
1073
1074 /// Shared settle-call logging: a `409` means the server already considers
1075 /// the task lost or settled (our outcome is dropped, its reassignment is
1076 /// authoritative); a transport failure means the task returns to the
1077 /// queue when the lease expires.
1078 async fn log_settle(&self, call: &str, response: reqwest::Result<reqwest::Response>) {
1079 match response {
1080 Ok(response) if response.status() == StatusCode::OK => {
1081 tracing::debug!("{call} accepted");
1082 }
1083 Ok(response) if response.status() == StatusCode::CONFLICT => {
1084 tracing::warn!(
1085 "{call} rejected (409): task already lost or settled; \
1086 outcome dropped in favor of the server's reassignment"
1087 );
1088 }
1089 Ok(response) => {
1090 tracing::warn!(status = %response.status(), "{call} call failed");
1091 }
1092 Err(e) => {
1093 tracing::error!(
1094 error = %e,
1095 "{call} call failed; the task returns to the queue at lease expiry"
1096 );
1097 }
1098 }
1099 }
1100}
1101
1102#[cfg(test)]
1103mod tests {
1104 use super::*;
1105 use serde_json::json;
1106
1107 #[test]
1108 fn closure_and_arc_implement_activity() {
1109 let closure = |ctx: ActivityContext| async move { Ok(ctx.payload().clone()) };
1110 fn assert_activity<T: Activity>(_: &T) {}
1111 assert_activity(&closure);
1112
1113 let arc: Arc<dyn Activity> = Arc::new(closure);
1114 assert_activity(&arc);
1115 }
1116
1117 #[tokio::test]
1118 async fn run_with_receipt_defaults_to_no_receipt() {
1119 // Pre-receipt activities (and the closure blanket impl) complete
1120 // through the provided method unchanged: their result is preserved
1121 // and no receipt is invented on their behalf.
1122 let closure = |ctx: ActivityContext| async move { Ok(json!({"echo": ctx.payload()})) };
1123 let ctx = ActivityContext {
1124 task_id: "t-1".into(),
1125 kind: "k".into(),
1126 attempt: 1,
1127 idempotency_key: None,
1128 payload: json!({"n": 1}),
1129 };
1130 let completion = closure.run_with_receipt(ctx).await.unwrap();
1131 assert_eq!(completion.result, json!({"echo": {"n": 1}}));
1132 assert!(completion.receipt.is_none());
1133 }
1134
1135 #[tokio::test]
1136 async fn run_with_receipt_carries_the_reported_receipt() {
1137 struct Charger;
1138 #[async_trait]
1139 impl Activity for Charger {
1140 async fn run(&self, _ctx: ActivityContext) -> Result<Value> {
1141 unreachable!("the worker drives run_with_receipt")
1142 }
1143 async fn run_with_receipt(&self, ctx: ActivityContext) -> Result<ActivityCompletion> {
1144 Ok(ActivityCompletion {
1145 result: json!({"charged": true}),
1146 receipt: Some(EffectReceipt {
1147 provider: "stripe".into(),
1148 provider_id: "ch_3PKd".into(),
1149 idempotency_key: ctx.idempotency_key().unwrap_or_default().to_owned(),
1150 task_id: Some(ctx.task_id().to_owned()),
1151 // R0.7: the typed-kernel effect id is unset on this
1152 // pre-kernel receipt path; the idempotency-key lookup
1153 // keeps serving it.
1154 effect_id: None,
1155 }),
1156 })
1157 }
1158 }
1159 let ctx = ActivityContext {
1160 task_id: "t-7".into(),
1161 kind: "charge_card".into(),
1162 attempt: 1,
1163 idempotency_key: Some("run-9:charge:7".into()),
1164 payload: json!({}),
1165 };
1166 let completion = Charger.run_with_receipt(ctx).await.unwrap();
1167 let receipt = completion.receipt.expect("receipt reported");
1168 assert_eq!(receipt.provider_id, "ch_3PKd");
1169 assert_eq!(receipt.idempotency_key, "run-9:charge:7");
1170 assert_eq!(receipt.task_id.as_deref(), Some("t-7"));
1171 }
1172
1173 #[tokio::test]
1174 async fn arc_dyn_activity_delegates_run_with_receipt() {
1175 // Regression: the worker stores every registered handler as
1176 // `Arc<dyn Activity>` and drives it through `run_with_receipt`.
1177 // When the `Arc<dyn Activity>` impl overrode only `run`, the
1178 // default `run_with_receipt` answered for the Arc and silently
1179 // dropped the receipt — the crash-recovery proof caught it.
1180 struct Charger;
1181 #[async_trait]
1182 impl Activity for Charger {
1183 async fn run(&self, _ctx: ActivityContext) -> Result<Value> {
1184 unreachable!("the worker drives run_with_receipt")
1185 }
1186 async fn run_with_receipt(&self, ctx: ActivityContext) -> Result<ActivityCompletion> {
1187 Ok(ActivityCompletion {
1188 result: json!({"charged": true}),
1189 receipt: Some(EffectReceipt {
1190 provider: "stripe".into(),
1191 provider_id: "ch_arc".into(),
1192 idempotency_key: ctx.idempotency_key().unwrap_or_default().to_owned(),
1193 task_id: Some(ctx.task_id().to_owned()),
1194 // R0.7: the typed-kernel effect id is unset on this
1195 // pre-kernel receipt path; the idempotency-key lookup
1196 // keeps serving it.
1197 effect_id: None,
1198 }),
1199 })
1200 }
1201 }
1202 let activity: Arc<dyn Activity> = Arc::new(Charger);
1203 let ctx = ActivityContext {
1204 task_id: "t-9".into(),
1205 kind: "charge_card".into(),
1206 attempt: 2,
1207 idempotency_key: Some("run-1:charge:9".into()),
1208 payload: json!({}),
1209 };
1210 let completion = activity.run_with_receipt(ctx).await.unwrap();
1211 assert_eq!(completion.result, json!({"charged": true}));
1212 let receipt = completion
1213 .receipt
1214 .expect("the Arc must delegate the receipt");
1215 assert_eq!(receipt.provider_id, "ch_arc");
1216 assert_eq!(receipt.idempotency_key, "run-1:charge:9");
1217 }
1218
1219 #[test]
1220 fn classify_error_maps_to_the_shared_taxonomy() {
1221 // The transient executor classes are upstream dependency failures.
1222 assert_eq!(
1223 classify_error(&RustyError::Llm("rate limited".into())),
1224 (ErrorClass::DependencyFailure, true)
1225 );
1226 assert_eq!(
1227 classify_error(&RustyError::Tool("backend exploded".into())),
1228 (ErrorClass::DependencyFailure, true)
1229 );
1230 // Contract violations are invalid input.
1231 assert_eq!(
1232 classify_error(&RustyError::Graph("cycle".into())),
1233 (ErrorClass::InvalidInput, false)
1234 );
1235 assert_eq!(
1236 classify_error(&RustyError::InvalidUpdate("double write".into())),
1237 (ErrorClass::InvalidInput, false)
1238 );
1239 // Everything else is unclassified and hard.
1240 assert_eq!(
1241 classify_error(&RustyError::Node("boom".into())),
1242 (ErrorClass::Unknown, false)
1243 );
1244 assert_eq!(
1245 classify_error(&RustyError::Checkpoint("io".into())),
1246 (ErrorClass::Unknown, false)
1247 );
1248 assert_eq!(
1249 classify_error(&RustyError::Replay("diverged".into())),
1250 (ErrorClass::Unknown, false)
1251 );
1252 let serde_err = serde_json::from_str::<Value>("{").unwrap_err();
1253 assert_eq!(
1254 classify_error(&RustyError::Serialization(serde_err)),
1255 (ErrorClass::Unknown, false)
1256 );
1257 // Interrupts have no suspend semantics here; they settle as a
1258 // non-retryable cancellation.
1259 assert_eq!(
1260 classify_error(&RustyError::Interrupt { value: json!(null) }),
1261 (ErrorClass::Cancelled, false)
1262 );
1263 }
1264
1265 #[test]
1266 fn claimed_task_decodes_the_server_wire_shape() {
1267 // The claim response wraps the task record: `{"task": {…}}`. Fields
1268 // the worker does not model (pool, status, lease, timestamps) must
1269 // be tolerated.
1270 let body = json!({
1271 "task": {
1272 "task_id": "t-1",
1273 "kind": "send_email",
1274 "payload": {"to": "a@b.c"},
1275 "pool": "default",
1276 "status": "leased",
1277 "attempt": 2,
1278 "max_attempts": 3,
1279 "error_class": null,
1280 "last_error": "previous attempt blew up",
1281 "idempotency_key": "run-9:charge:7",
1282 "result": null,
1283 "run_id": "run-9",
1284 "thread_id": "thread-1",
1285 "cancel_requested": false,
1286 "deadline": "2026-08-07T13:00:00Z",
1287 "lease": {"owner": "w-1", "expires_at": "2026-08-07T12:00:30Z"},
1288 "next_attempt_at": null,
1289 "created_at": "2026-08-07T12:00:00Z",
1290 "updated_at": "2026-08-07T12:00:01Z"
1291 }
1292 });
1293 let claimed: ClaimResponse = serde_json::from_value(body).unwrap();
1294 let task = claimed.task;
1295 assert_eq!(task.task_id, "t-1");
1296 assert_eq!(task.kind, "send_email");
1297 assert_eq!(task.payload, json!({"to": "a@b.c"}));
1298 assert_eq!(task.attempt, 2);
1299 assert_eq!(task.max_attempts, 3);
1300 assert_eq!(task.idempotency_key.as_deref(), Some("run-9:charge:7"));
1301 assert_eq!(task.run_id.as_deref(), Some("run-9"));
1302 assert_eq!(task.thread_id.as_deref(), Some("thread-1"));
1303 assert!(!task.cancel_requested);
1304 assert_eq!(
1305 task.deadline,
1306 DateTime::parse_from_rfc3339("2026-08-07T13:00:00Z")
1307 .ok()
1308 .map(|dt| dt.with_timezone(&Utc))
1309 );
1310 }
1311
1312 #[test]
1313 fn claimed_task_defaults_optional_fields() {
1314 // A pre-wave-2 server omits the cancellation fields entirely.
1315 let task: ClaimedTask =
1316 serde_json::from_value(json!({"task_id": "t-1", "kind": "k"})).unwrap();
1317 assert_eq!(task.payload, Value::Null);
1318 assert_eq!(task.attempt, 0);
1319 assert!(task.idempotency_key.is_none());
1320 assert!(task.run_id.is_none() && task.thread_id.is_none());
1321 assert!(!task.cancel_requested);
1322 assert!(task.deadline.is_none());
1323 }
1324
1325 #[test]
1326 fn claim_request_omits_pools_when_unconfigured() {
1327 let body = ClaimRequest {
1328 worker_id: "w-1",
1329 pools: None,
1330 worker_version: None,
1331 lease_ms: 100,
1332 };
1333 let value = serde_json::to_value(&body).unwrap();
1334 assert_eq!(value, json!({"worker_id": "w-1", "lease_ms": 100}));
1335
1336 let pools = vec!["gpu".to_string()];
1337 let body = ClaimRequest {
1338 worker_id: "w-1",
1339 pools: Some(pools.as_slice()),
1340 worker_version: None,
1341 lease_ms: 100,
1342 };
1343 let value = serde_json::to_value(&body).unwrap();
1344 assert_eq!(
1345 value,
1346 json!({"worker_id": "w-1", "pools": ["gpu"], "lease_ms": 100})
1347 );
1348 }
1349
1350 #[test]
1351 fn claim_request_carries_the_advertised_worker_version() {
1352 // Wave 3: a configured version travels on the claim; unconfigured
1353 // the field is omitted entirely (pre-wave-3 servers never see it).
1354 let body = ClaimRequest {
1355 worker_id: "w-1",
1356 pools: None,
1357 worker_version: Some("activity-worker/1.4.0"),
1358 lease_ms: 100,
1359 };
1360 let value = serde_json::to_value(&body).unwrap();
1361 assert_eq!(
1362 value,
1363 json!({"worker_id": "w-1", "worker_version": "activity-worker/1.4.0", "lease_ms": 100})
1364 );
1365 }
1366
1367 #[test]
1368 fn settle_request_shapes_match_the_contract() {
1369 let heartbeat = HeartbeatRequest {
1370 worker_id: "w-1",
1371 lease_ms: 100,
1372 };
1373 assert_eq!(
1374 serde_json::to_value(&heartbeat).unwrap(),
1375 json!({"worker_id": "w-1", "lease_ms": 100})
1376 );
1377
1378 let result = json!({"sent": true});
1379 let complete = CompleteRequest {
1380 worker_id: "w-1",
1381 result: &result,
1382 receipt: None,
1383 };
1384 assert_eq!(
1385 serde_json::to_value(&complete).unwrap(),
1386 json!({"worker_id": "w-1", "result": {"sent": true}})
1387 );
1388
1389 // An activity reporting a receipt carries it with the completion;
1390 // without one the body is byte-identical to the pre-receipt shape.
1391 let receipt = EffectReceipt {
1392 provider: "stripe".into(),
1393 provider_id: "ch_3PKd".into(),
1394 idempotency_key: "run-9:charge:7".into(),
1395 task_id: Some("t-1".into()),
1396 effect_id: None,
1397 };
1398 let complete = CompleteRequest {
1399 worker_id: "w-1",
1400 result: &result,
1401 receipt: Some(&receipt),
1402 };
1403 assert_eq!(
1404 serde_json::to_value(&complete).unwrap(),
1405 json!({
1406 "worker_id": "w-1",
1407 "result": {"sent": true},
1408 "receipt": {
1409 "provider": "stripe",
1410 "provider_id": "ch_3PKd",
1411 "idempotency_key": "run-9:charge:7",
1412 "task_id": "t-1"
1413 }
1414 })
1415 );
1416
1417 let fail = FailRequest {
1418 worker_id: "w-1",
1419 error_class: ErrorClass::DependencyFailure,
1420 message: "backend exploded",
1421 retryable: true,
1422 };
1423 assert_eq!(
1424 serde_json::to_value(&fail).unwrap(),
1425 json!({
1426 "worker_id": "w-1",
1427 "error_class": "dependency_failure",
1428 "message": "backend exploded",
1429 "retryable": true
1430 })
1431 );
1432 }
1433
1434 #[test]
1435 fn heartbeat_and_fail_responses_decode() {
1436 let heartbeat: HeartbeatResponse =
1437 serde_json::from_value(json!({"lease_expires_at": "2026-08-07T12:00:30Z"})).unwrap();
1438 assert_eq!(
1439 heartbeat.lease_expires_at.as_deref(),
1440 Some("2026-08-07T12:00:30Z")
1441 );
1442 // The cancellation hint defaults off (pre-wave-2 servers omit it).
1443 assert!(!heartbeat.cancel_requested);
1444
1445 let heartbeat: HeartbeatResponse = serde_json::from_value(
1446 json!({"lease_expires_at": "2026-08-07T12:00:30Z", "cancel_requested": true}),
1447 )
1448 .unwrap();
1449 assert!(heartbeat.cancel_requested);
1450
1451 let fail: FailResponse = serde_json::from_value(json!({
1452 "requeued": true,
1453 "next_attempt_at": "2026-08-07T12:01:00Z",
1454 "dead": false
1455 }))
1456 .unwrap();
1457 assert!(fail.requeued && !fail.dead);
1458 assert_eq!(
1459 fail.next_attempt_at.as_deref(),
1460 Some("2026-08-07T12:01:00Z")
1461 );
1462 }
1463
1464 #[test]
1465 fn lease_is_clamped_to_the_server_bounds() {
1466 let worker = ActivityWorker::new("http://127.0.0.1:1").with_lease(Duration::from_millis(5));
1467 assert_eq!(worker.lease, MIN_LEASE);
1468 let worker =
1469 ActivityWorker::new("http://127.0.0.1:1").with_lease(Duration::from_secs(86_400));
1470 assert_eq!(worker.lease, MAX_LEASE);
1471 }
1472}