Skip to main content

rusty_worker/
activity.rs

1//! Durable activities (R0.6): [`ActivityWorker`], the leased-task execution
2//! loop against the rusty-agent-server task queue.
3//!
4//! Where [`crate::serve`] answers one-shot `/execute` calls pushed by a
5//! [`RemoteNode`](rusty_agent_runtime::remote::RemoteNode), an
6//! `ActivityWorker` *pulls* work: it claims tasks from the server under a
7//! lease, renews that lease with heartbeats while the handler runs, and
8//! settles the task with a complete or fail call. The server re-queues tasks
9//! whose lease expires, so a crashed worker never strands work. The promise
10//! is *effectively-once* execution when activity side effects are idempotent
11//! — not universal exactly-once.
12//!
13//! ## The task model
14//!
15//! A task is a generic unit of durable work: a `kind` label plus an
16//! arbitrary JSON `payload`, enqueued via `POST /tasks` and claimed by pool.
17//! The worker registers one [`Activity`] handler per `kind`; the handler
18//! receives an [`ActivityContext`] (task id, attempt, idempotency key,
19//! payload) and returns the JSON result stored on the task record.
20//!
21//! ## The lease protocol (client side)
22//!
23//! - `POST {base}/tasks/claim` — body `{worker_id, pools?, worker_version?,
24//!   lease_ms}`; a `200` carries `{"task": {…}}` (a [`ClaimedTask`]), a `204`
25//!   means no work is available in the claimed pools. `worker_version` (R0.6
26//!   wave 3) is the worker's advertised version: the server matches it
27//!   exactly against a task's version pin, so a run pinned to a version
28//!   keeps dispatching to workers running it until the run finishes.
29//! - `POST {base}/tasks/{task_id}/heartbeat` — body `{worker_id, lease_ms}`,
30//!   sent from a background task every `lease / 3`; a `200` renews the lease
31//!   (`{lease_expires_at, cancel_requested}`), a `409` means the lease is
32//!   lost.
33//! - `POST {base}/tasks/{task_id}/complete` — body `{worker_id, result}`,
34//!   where `result` is the handler's JSON return value; an activity that
35//!   performed an idempotent effect additionally carries its `receipt`
36//!   (the provider's confirmation — see [`Activity::run_with_receipt`]),
37//!   which the server journals into the task's run as an `effect_receipt`
38//!   event.
39//! - `POST {base}/tasks/{task_id}/fail` — body
40//!   `{worker_id, error_class, message, retryable}`; the reply
41//!   `{requeued, next_attempt_at, dead}` is logged for operators.
42//!
43//! A `409` from any of these calls means the server considers the task lost
44//! or already settled; the worker then abandons the activity without further
45//! calls, so the server's reassignment stays the single source of truth.
46//!
47//! ## Semantics
48//!
49//! - **One activity in flight.** The loop claims, executes, and settles one
50//!   task at a time. Run several `ActivityWorker` instances for parallelism;
51//!   pool routing ([`ActivityWorker::with_pools`]) keeps them on disjoint
52//!   queues.
53//! - **Lease loss aborts execution.** When a heartbeat answers `409`, the
54//!   handler future is aborted (dropped at its next yield point) and the
55//!   worker makes no settle call — the server will reassign the task.
56//!   Handler code must tolerate being abandoned mid-effect; that is the
57//!   effectively-once contract, and it is why activity side effects should
58//!   be keyed by [`ActivityContext::task_id`] /
59//!   [`ActivityContext::idempotency_key`].
60//! - **Cancellation aborts promptly, and reports.** When a heartbeat
61//!   carries `cancel_requested: true` — or the task's whole-task
62//!   `deadline` passes mid-attempt — the handler is aborted the same way,
63//!   but the worker *does* settle: it reports the attempt as
64//!   [`ErrorClass::Cancelled`] through `/fail`, which the server's shared
65//!   retry policy fails outright, so the record ends terminal-cancelled —
66//!   never retried, never dead-lettered. A task claimed with the deadline
67//!   already passed (or the cancel flag already set) is reported cancelled
68//!   without running the handler at all.
69//! - **Graceful drain (R0.6 wave 2c).** Cancelling the shutdown
70//!   [`CancellationToken`] *is* the drain request — idempotent, and
71//!   race-safe against an in-flight claim poll (the poll itself is
72//!   `select!`ed against the token). Draining stops claiming immediately;
73//!   an in-flight activity keeps heartbeating and settles normally,
74//!   bounded by [`ActivityWorker::with_drain_grace`] (default
75//!   [`DEFAULT_DRAIN_GRACE`], 25 s — under the 30 s default lease and
76//!   Kubernetes' 30 s pod-termination grace). An attempt that outlives
77//!   the grace is aborted and deliberately **not** settled: reporting it
78//!   [`ErrorClass::Cancelled`] would kill the task (cancelled is
79//!   terminal), while leaving it unsettled releases it back to visibility
80//!   at lease expiry for another worker — the design's "finishes or
81//!   fast-fails in-flight attempts within a grace period, and releases
82//!   the rest".
83//! - **Failure classification.** Handler errors reach `/fail` under the
84//!   frozen [`ErrorClass`] taxonomy (shared with the server scheduler via
85//!   `rusty_agent_runtime::durable`), with the `retryable` flag mirroring
86//!   the executor's judgment: `Llm` and `Tool` failures are the transient
87//!   classes; everything else is reported non-retryable.
88//!
89//! ```no_run
90//! use std::time::Duration;
91//! use rusty_worker::{ActivityContext, ActivityWorker};
92//! use tokio_util::sync::CancellationToken;
93//!
94//! # async fn demo() {
95//! let worker = ActivityWorker::new("http://127.0.0.1:8080")
96//!     .register("send_receipt", |ctx: ActivityContext| async move {
97//!         // `task_id` / `idempotency_key` are the side-effect correlation
98//!         // handles that make redelivery effectively-once.
99//!         let _dedup_key = ctx.idempotency_key().unwrap_or_else(|| ctx.task_id());
100//!         let to = ctx.payload()["to"].as_str().unwrap_or("unknown");
101//!         // … perform the effect …
102//!         Ok(serde_json::json!({"sent": true, "to": to}))
103//!     })
104//!     .with_worker_id("email-worker-1")
105//!     .with_lease(Duration::from_secs(30))
106//!     .with_pools(["email"]);
107//!
108//! // Cancel the token (e.g. from a SIGTERM handler) to drain and exit.
109//! worker.run(CancellationToken::new()).await;
110//! # }
111//! ```
112
113use std::collections::HashMap;
114use std::future::Future;
115use std::sync::Arc;
116use std::time::Duration;
117
118use async_trait::async_trait;
119use chrono::{DateTime, Utc};
120use reqwest::StatusCode;
121use rusty_agent_runtime::durable::ErrorClass;
122use rusty_agent_runtime::error::{Result, RustyError};
123use rusty_agent_runtime::record::EffectReceipt;
124use serde::{Deserialize, Serialize};
125use serde_json::Value;
126use tokio_util::sync::CancellationToken;
127use tracing::Instrument;
128use uuid::Uuid;
129
130/// Default lease requested on claim and renewed by heartbeats (30 s).
131pub const DEFAULT_LEASE: Duration = Duration::from_secs(30);
132
133/// Default per-request HTTP timeout for claim/heartbeat/settle calls (10 s).
134pub const DEFAULT_REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
135
136/// Default base delay between claim polls when no work is available (or the
137/// server is unreachable); consecutive empty polls back off exponentially up
138/// to [`DEFAULT_CLAIM_BACKOFF_MAX`].
139pub const DEFAULT_CLAIM_BACKOFF_BASE: Duration = Duration::from_millis(100);
140
141/// Default cap for the claim-poll backoff.
142pub const DEFAULT_CLAIM_BACKOFF_MAX: Duration = Duration::from_secs(5);
143
144/// Default drain grace (25 s): once draining starts, an in-flight activity
145/// gets this long to finish and settle before the worker aborts it and
146/// exits, leaving the attempt for the server to reassign at lease expiry.
147///
148/// Chosen to sit under the [`DEFAULT_LEASE`] of 30 s — the drain never
149/// outlives the visibility window the lease already provides — and under
150/// Kubernetes' default 30 s pod-termination grace, so a drained worker
151/// exits before SIGKILL would arrive. Override with
152/// [`ActivityWorker::with_drain_grace`]; when you also change the lease,
153/// keep the grace below it.
154pub const DEFAULT_DRAIN_GRACE: Duration = Duration::from_secs(25);
155
156// The server's accepted `lease_ms` range (`tasks::MIN_LEASE_MS..=MAX_LEASE_MS`
157// in rusty-agent-server): below 100 ms a lease is an instant expiry, above one
158// hour a lost worker strands its task for too long. `with_lease` clamps to
159// this range so a misconfigured worker gets a sane lease instead of a 400
160// on every claim.
161const MIN_LEASE: Duration = Duration::from_millis(100);
162const MAX_LEASE: Duration = Duration::from_secs(3_600);
163
164/// The input to every activity invocation: the claimed task's identity,
165/// attempt ordinal, dedup key, and payload.
166///
167/// `task_id` (and, when the enqueue side set one, `idempotency_key`) are
168/// the correlation handles for external side effects — a redelivered task
169/// (lease expiry, crash) re-runs the handler with the same ids, which is
170/// what makes the effect effectively-once.
171#[derive(Debug, Clone)]
172pub struct ActivityContext {
173    task_id: String,
174    kind: String,
175    attempt: u32,
176    idempotency_key: Option<String>,
177    payload: Value,
178}
179
180impl ActivityContext {
181    /// The server-minted task id.
182    pub fn task_id(&self) -> &str {
183        &self.task_id
184    }
185
186    /// The task kind this invocation was dispatched under.
187    pub fn kind(&self) -> &str {
188        &self.kind
189    }
190
191    /// The 1-based attempt ordinal (`1` on first delivery, higher after a
192    /// requeue).
193    pub fn attempt(&self) -> u32 {
194        self.attempt
195    }
196
197    /// The dedup key supplied at enqueue time, if any.
198    pub fn idempotency_key(&self) -> Option<&str> {
199        self.idempotency_key.as_deref()
200    }
201
202    /// The task payload, exactly as enqueued.
203    pub fn payload(&self) -> &Value {
204        &self.payload
205    }
206}
207
208/// An activity: one durable unit of work, dispatched by task `kind`.
209///
210/// Implement this directly for stateful activities, or just pass an async
211/// closure `Fn(ActivityContext) -> impl Future<Output = Result<Value>>` to
212/// [`ActivityWorker::register`] — a blanket impl covers it, mirroring the
213/// [`Node`](rusty_agent_runtime::node::Node) ergonomics.
214///
215/// The returned `Value` is stored verbatim as the task's `result`. An
216/// `Err` is classified into the shared [`ErrorClass`] taxonomy and reported
217/// to `POST /tasks/{id}/fail`.
218///
219/// Activities performing an `Idempotent` effect should override
220/// [`Activity::run_with_receipt`] instead: the receipt it returns — the
221/// provider's own confirmation, under the task's idempotency key — travels
222/// with the completion and is journaled into the task's run as evidence
223/// exact replay can serve instead of re-sending the effect.
224#[async_trait]
225pub trait Activity: Send + Sync {
226    /// Execute the activity against a claimed task.
227    async fn run(&self, ctx: ActivityContext) -> Result<Value>;
228
229    /// Execute the activity, additionally reporting an effect receipt with
230    /// the result. The default wraps [`Activity::run`] with no receipt, so
231    /// existing activities (and the closure blanket impl) are unaffected —
232    /// a receipt is something an activity opts into reporting, never
233    /// something the worker invents on its behalf: only the code that
234    /// performed the effect knows the provider's confirmation id.
235    async fn run_with_receipt(&self, ctx: ActivityContext) -> Result<ActivityCompletion> {
236        Ok(ActivityCompletion {
237            result: self.run(ctx).await?,
238            receipt: None,
239        })
240    }
241}
242
243/// What an activity reports on success: the stored `result` plus, when the
244/// activity performed an `Idempotent` effect, its [`EffectReceipt`] — the
245/// effect's own confirmation (provider id, idempotency key), which the
246/// server journals into the task's run as a `effect_receipt` event.
247#[derive(Debug, Clone)]
248pub struct ActivityCompletion {
249    /// The value stored as the task's `result` (any JSON).
250    pub result: Value,
251
252    /// The effect's confirmation, when the activity performed an idempotent
253    /// effect and the provider confirmed it. `None` for pure/read-only work
254    /// and for effects that carry no confirmation handle.
255    pub receipt: Option<EffectReceipt>,
256}
257
258/// Blanket implementation for async closures/functions:
259/// `Fn(ActivityContext) -> impl Future<Output = Result<Value>>`.
260#[async_trait]
261impl<F, Fut> Activity for F
262where
263    F: Fn(ActivityContext) -> Fut + Send + Sync,
264    Fut: Future<Output = Result<Value>> + Send,
265{
266    async fn run(&self, ctx: ActivityContext) -> Result<Value> {
267        (self)(ctx).await
268    }
269}
270
271/// Allow `Arc<dyn Activity>` itself to be registered (useful for sharing one
272/// activity implementation across workers).
273///
274/// Both methods delegate: overriding only `run` would make the default
275/// `run_with_receipt` answer for the `Arc` — wrapping the inner `run` and
276/// silently dropping the receipt a receipt-carrying activity reported. The
277/// worker stores every registered handler as `Arc<dyn Activity>` and drives
278/// it through `run_with_receipt`, so this delegation is what carries the
279/// effect receipt from the handler to the completion call.
280#[async_trait]
281impl Activity for Arc<dyn Activity> {
282    async fn run(&self, ctx: ActivityContext) -> Result<Value> {
283        self.as_ref().run(ctx).await
284    }
285
286    async fn run_with_receipt(&self, ctx: ActivityContext) -> Result<ActivityCompletion> {
287        self.as_ref().run_with_receipt(ctx).await
288    }
289}
290
291/// The task a worker receives on a successful claim: the fields of the
292/// server's task record a worker needs (`POST /tasks/claim` →
293/// `{"task": {…}}`). Fields the worker does not consume are ignored on
294/// deserialization, so additive server-side evolution keeps working.
295#[derive(Debug, Clone, Deserialize)]
296pub struct ClaimedTask {
297    /// The server-minted task id, used to address heartbeat/complete/fail.
298    pub task_id: String,
299
300    /// The dispatch label the worker's handler registry is keyed by.
301    pub kind: String,
302
303    /// The work payload, exactly as enqueued.
304    #[serde(default)]
305    pub payload: Value,
306
307    /// The 1-based attempt ordinal (higher after a requeue).
308    #[serde(default)]
309    pub attempt: u32,
310
311    /// The attempt ceiling configured at enqueue time.
312    #[serde(default)]
313    pub max_attempts: u32,
314
315    /// The dedup key supplied at enqueue time, if any.
316    #[serde(default)]
317    pub idempotency_key: Option<String>,
318
319    /// Run linkage (reserved for the run-side outbox wave); carried into the
320    /// activity's tracing span when present.
321    #[serde(default)]
322    pub run_id: Option<String>,
323
324    /// See [`ClaimedTask::run_id`].
325    #[serde(default)]
326    pub thread_id: Option<String>,
327
328    /// Cancellation signalled before the claim was handed out. A
329    /// well-behaved server never leases such a task (its claim path
330    /// finalizes them as cancelled), but the worker defends anyway: a
331    /// claimed task with this flag is reported cancelled without running
332    /// the handler.
333    #[serde(default)]
334    pub cancel_requested: bool,
335
336    /// Whole-task deadline, across attempts. The worker refuses to start
337    /// an attempt past it and aborts a running attempt at it, reporting
338    /// [`ErrorClass::Cancelled`] — deadline expiry is cancellation by
339    /// clock, per the Durable Work design.
340    #[serde(default)]
341    pub deadline: Option<DateTime<Utc>>,
342}
343
344/// `POST /tasks/claim` request body.
345#[derive(Debug, Serialize)]
346struct ClaimRequest<'a> {
347    worker_id: &'a str,
348    #[serde(skip_serializing_if = "Option::is_none")]
349    pools: Option<&'a [String]>,
350    /// The worker's advertised version (R0.6 wave 3), matched exactly
351    /// against a task's `worker_version` pin. Omitted (not null) when
352    /// unconfigured — an unversioned worker takes unpinned work only, and
353    /// pre-wave-3 servers never see the field.
354    #[serde(skip_serializing_if = "Option::is_none")]
355    worker_version: Option<&'a str>,
356    lease_ms: u64,
357}
358
359/// `POST /tasks/claim` success body: the task, wrapped.
360#[derive(Debug, Deserialize)]
361struct ClaimResponse {
362    task: ClaimedTask,
363}
364
365/// `POST /tasks/{id}/heartbeat` request body.
366#[derive(Debug, Serialize)]
367struct HeartbeatRequest<'a> {
368    worker_id: &'a str,
369    lease_ms: u64,
370}
371
372/// `POST /tasks/{id}/heartbeat` success body. `lease_expires_at` is an
373/// RFC 3339 timestamp; the worker treats it as informational (the server's
374/// `409` is the authoritative lease-loss signal) and logs it.
375/// `cancel_requested` is the cancellation hint: on `true` the worker
376/// aborts the in-flight attempt and reports it [`ErrorClass::Cancelled`].
377/// Absent on pre-wave-2 servers, hence the default.
378#[derive(Debug, Deserialize)]
379struct HeartbeatResponse {
380    lease_expires_at: Option<String>,
381    #[serde(default)]
382    cancel_requested: bool,
383}
384
385/// `POST /tasks/{id}/complete` request body. `receipt` is omitted (not
386/// null) when the activity reports none, keeping the body identical for
387/// pre-receipt activities.
388#[derive(Debug, Serialize)]
389struct CompleteRequest<'a> {
390    worker_id: &'a str,
391    result: &'a Value,
392    #[serde(skip_serializing_if = "Option::is_none")]
393    receipt: Option<&'a EffectReceipt>,
394}
395
396/// `POST /tasks/{id}/fail` request body. `error_class` serializes to the
397/// frozen snake_case wire spelling of [`ErrorClass`].
398#[derive(Debug, Serialize)]
399struct FailRequest<'a> {
400    worker_id: &'a str,
401    error_class: ErrorClass,
402    message: &'a str,
403    retryable: bool,
404}
405
406/// `POST /tasks/{id}/fail` success body: the server's retry decision.
407#[derive(Debug, Deserialize)]
408struct FailResponse {
409    requeued: bool,
410    next_attempt_at: Option<String>,
411    dead: bool,
412}
413
414/// Classify a handler error for the `/fail` call.
415///
416/// Two independent judgments, both owned by the executor of the work:
417///
418/// - `error_class` — the frozen [`ErrorClass`] taxonomy shared with the
419///   server scheduler. `Llm` and `Tool` failures are upstream
420///   ([`ErrorClass::DependencyFailure`]); `Graph` and `InvalidUpdate` are
421///   contract violations ([`ErrorClass::InvalidInput`]); everything
422///   unclassifiable is [`ErrorClass::Unknown`]. An `Interrupt` error is
423///   [`ErrorClass::Cancelled`]: the task-queue protocol has no suspend
424///   semantics (HITL wiring is the run-outbox wave's concern), so a handler
425///   interrupt settles as a non-retryable cancellation.
426/// - `retryable` — the executor's own taxonomy: `Llm` and `Tool` are the
427///   transient classes; everything else is a hard failure.
428fn classify_error(error: &RustyError) -> (ErrorClass, bool) {
429    match error {
430        RustyError::Llm(_) => (ErrorClass::DependencyFailure, true),
431        RustyError::Tool(_) => (ErrorClass::DependencyFailure, true),
432        RustyError::Node(_) => (ErrorClass::Unknown, false),
433        RustyError::Graph(_) => (ErrorClass::InvalidInput, false),
434        RustyError::InvalidUpdate(_) => (ErrorClass::InvalidInput, false),
435        RustyError::Checkpoint(_) => (ErrorClass::Unknown, false),
436        RustyError::Replay(_) => (ErrorClass::Unknown, false),
437        RustyError::Serialization(_) => (ErrorClass::Unknown, false),
438        RustyError::Interrupt { .. } => (ErrorClass::Cancelled, false),
439        // A handler whose own inner run was drained propagates the
440        // cancellation class: control flow, never retried.
441        RustyError::Cancelled(_) => (ErrorClass::Cancelled, false),
442    }
443}
444
445/// The outcome of one claim poll.
446enum ClaimOutcome {
447    /// A task was claimed and must be executed and settled. Boxed: the
448    /// claimed-task payload dwarfs the other variants and the wasm-enabled
449    /// feature combination pushes it past clippy's size threshold.
450    Task(Box<ClaimedTask>),
451    /// The server answered `204`: no work available.
452    Empty,
453    /// The poll failed (transport error, unexpected status, undecodable
454    /// body); the caller backs off and polls again.
455    Unavailable,
456}
457
458/// Why the attempt runner stopped waiting on the handler future: finished
459/// normally, the lease is gone (abandon — the server reassigns), or the
460/// task was cancelled (abort, then report [`ErrorClass::Cancelled`]
461/// through the fail path while we still hold the lease), or the drain
462/// grace expired (abort, then leave the attempt unsettled — see
463/// [`ActivityWorker::run`]'s drain semantics).
464enum AttemptOutcome {
465    /// The handler future completed (possibly with an error or panic).
466    Finished(std::result::Result<Result<ActivityCompletion>, tokio::task::JoinError>),
467    /// A heartbeat answered `409`: the server considers the task lost or
468    /// settled. The worker makes no further calls for it.
469    LeaseLost,
470    /// Cancellation reached the worker — a heartbeat carried
471    /// `cancel_requested: true`, or the whole-task deadline expired. The
472    /// payload is the reason recorded on the fail report.
473    Cancelled(&'static str),
474    /// The worker is draining and the in-flight attempt outlived the
475    /// configured grace. Like [`AttemptOutcome::LeaseLost`] the attempt is
476    /// abandoned without a settle call — but deliberately, not because the
477    /// lease is gone: settling `cancelled` would kill the task outright,
478    /// while leaving it unsettled lets lease expiry hand it to a worker
479    /// that is not being deployed away.
480    DrainGraceExpired,
481}
482
483/// The deadline arm of the attempt runner's `select!`: fires when the
484/// whole-task deadline passes, or pends forever when the task has none —
485/// one arm shape either way.
486async fn sleep_until_deadline(until: Option<Duration>) {
487    match until {
488        Some(delay) => tokio::time::sleep(delay).await,
489        None => std::future::pending().await,
490    }
491}
492
493/// A durable worker that claims leased tasks from the rusty-agent-server task
494/// queue and executes them behind heartbeats.
495///
496/// This is the pull-based counterpart to [`crate::serve`]: instead of
497/// answering one-shot `/execute` calls, it polls `POST /tasks/claim`, runs
498/// the [`Activity`] registered for the claimed task's `kind`, renews the
499/// lease via a background heartbeat every `lease / 3`, and settles with
500/// `complete` or `fail` (classified error). See the
501/// [module documentation](self) for the full protocol and semantics.
502///
503/// Run several instances for parallelism; pool routing keeps them on
504/// disjoint queues.
505pub struct ActivityWorker {
506    handlers: HashMap<String, Arc<dyn Activity>>,
507    /// Server base URL, trailing slashes trimmed.
508    base_url: String,
509    /// The identity sent on every claim/heartbeat/settle call.
510    worker_id: String,
511    client: reqwest::Client,
512    /// Requested lease duration; heartbeats fire at `lease / 3`.
513    lease: Duration,
514    /// Pool names to claim from; empty means the server's default pool.
515    pools: Vec<String>,
516    /// The version advertised on every claim (R0.6 wave 3), matched
517    /// exactly against tasks' `worker_version` pins; `None` claims
518    /// unpinned work only.
519    worker_version: Option<String>,
520    claim_backoff_base: Duration,
521    claim_backoff_max: Duration,
522    /// How long an in-flight activity may run after draining starts before
523    /// the worker aborts it and exits (see [`DEFAULT_DRAIN_GRACE`]).
524    drain_grace: Duration,
525}
526
527impl ActivityWorker {
528    /// A worker claiming tasks from the rusty-agent-server at `base_url` (e.g.
529    /// `"http://127.0.0.1:8080"`).
530    ///
531    /// Register at least one [`Activity`] with [`ActivityWorker::register`]
532    /// before running. The worker id defaults to `worker-{uuid}`; set a
533    /// stable one with [`ActivityWorker::with_worker_id`] if operators
534    /// should recognize the worker across restarts.
535    pub fn new(base_url: impl Into<String>) -> Self {
536        Self {
537            handlers: HashMap::new(),
538            base_url: base_url.into().trim_end_matches('/').to_owned(),
539            worker_id: format!("worker-{}", Uuid::new_v4()),
540            // Same construction invariant as `RemoteNode`: the builder only
541            // sets a timeout and the rustls backend needs no platform
542            // initialization, so this cannot realistically fail.
543            client: reqwest::Client::builder()
544                .timeout(DEFAULT_REQUEST_TIMEOUT)
545                .build()
546                .expect("reqwest client builder with rustls must succeed"),
547            lease: DEFAULT_LEASE,
548            pools: Vec::new(),
549            worker_version: None,
550            claim_backoff_base: DEFAULT_CLAIM_BACKOFF_BASE,
551            claim_backoff_max: DEFAULT_CLAIM_BACKOFF_MAX,
552            drain_grace: DEFAULT_DRAIN_GRACE,
553        }
554    }
555
556    /// Register the activity executed for tasks of `kind`.
557    ///
558    /// Accepts any [`Activity`] implementation — including plain async
559    /// closures via the blanket impl. Registering the same kind twice
560    /// replaces the previous handler.
561    pub fn register<A>(mut self, kind: impl Into<String>, activity: A) -> Self
562    where
563        A: Activity + 'static,
564    {
565        self.handlers.insert(kind.into(), Arc::new(activity));
566        self
567    }
568
569    /// Override the worker identity sent on every call to the server.
570    pub fn with_worker_id(mut self, worker_id: impl Into<String>) -> Self {
571        self.worker_id = worker_id.into();
572        self
573    }
574
575    /// Override the requested lease duration. Heartbeats renew the lease
576    /// from a background task every `lease / 3` (clamped to ≥ 1 ms), so a
577    /// lease must comfortably outlive three heartbeat intervals of clock
578    /// drift and scheduling jitter — the [`DEFAULT_LEASE`] of 30 s is a good
579    /// floor for real deployments. The value is clamped to the server's
580    /// accepted range (100 ms – 1 h).
581    pub fn with_lease(mut self, lease: Duration) -> Self {
582        self.lease = lease.clamp(MIN_LEASE, MAX_LEASE);
583        self
584    }
585
586    /// Restrict claiming to the named pools (`"pools": [...]` on the claim
587    /// call). With no pools configured the claim omits the field and the
588    /// server's default pool applies.
589    pub fn with_pools<I, S>(mut self, pools: I) -> Self
590    where
591        I: IntoIterator<Item = S>,
592        S: Into<String>,
593    {
594        self.pools = pools.into_iter().map(Into::into).collect();
595        self
596    }
597
598    /// Advertise this worker's version on every claim (R0.6 wave 3). The
599    /// server matches it **exactly** against a task's `worker_version`
600    /// pin: tasks a run pinned to this version are leased to this worker
601    /// (and its version-mates) until the run finishes, so a mid-run deploy
602    /// never changes semantics under an in-flight execution; unpinned
603    /// tasks remain claimable regardless. With no version configured the
604    /// claim omits the field and the worker takes unpinned work only —
605    /// a worker that should serve pinned runs must advertise the version
606    /// it actually runs. Free-form string, but make it the real build
607    /// identity (crate version, image tag): a stale advertisement defeats
608    /// the pin's purpose.
609    pub fn with_worker_version(mut self, version: impl Into<String>) -> Self {
610        self.worker_version = Some(version.into());
611        self
612    }
613
614    /// Override the claim-poll backoff: empty or failed polls wait roughly
615    /// `base * 2^n` (capped at `max`) until a task is claimed.
616    pub fn with_claim_backoff(mut self, base: Duration, max: Duration) -> Self {
617        self.claim_backoff_base = base;
618        self.claim_backoff_max = max;
619        self
620    }
621
622    /// Override the drain grace (default [`DEFAULT_DRAIN_GRACE`]): how long
623    /// an in-flight activity may keep running after the shutdown token
624    /// fires before the worker aborts it and exits. An aborted attempt is
625    /// deliberately **not** settled — it returns to visibility at lease
626    /// expiry for another worker — so keep the grace under the lease:
627    /// beyond the lease the abort buys nothing (the server would reassign
628    /// at the same moment either way).
629    pub fn with_drain_grace(mut self, grace: Duration) -> Self {
630        self.drain_grace = grace;
631        self
632    }
633
634    /// The identity this worker presents to the server.
635    pub fn worker_id(&self) -> &str {
636        &self.worker_id
637    }
638
639    /// Run the claim → execute → settle loop until `shutdown` is cancelled,
640    /// then drain: stop claiming, settle any in-flight activity, and return.
641    ///
642    /// Cancelling `shutdown` **is** the drain request — there is no separate
643    /// `drain()` because the token already is the control: idempotent
644    /// (cancelling twice changes nothing), race-safe (a cancel racing an
645    /// in-flight claim poll still wins, because the claim itself is
646    /// `select!`ed against the token), and shareable (every clone of the
647    /// token observes the same signal). Wire it to SIGTERM/SIGINT in the
648    /// embedding binary; see the crate's `activity_worker_demo` example.
649    ///
650    /// Drain semantics: claiming stops immediately; an in-flight activity
651    /// keeps heartbeating and settles normally, bounded by
652    /// [`ActivityWorker::with_drain_grace`]. An attempt still running when
653    /// the grace expires is aborted and deliberately **not** settled —
654    /// reporting it [`ErrorClass::Cancelled`] would kill the task
655    /// (cancelled is terminal: never retried, never dead-lettered), which
656    /// is exactly what a deployment must not do. Left unsettled, the
657    /// attempt returns to visibility at lease expiry and another worker
658    /// picks it up — the design's "releases the rest, which return to
659    /// visibility for other workers".
660    ///
661    /// The loop never gives up on the server: claim/heartbeat/settle
662    /// transport failures are logged and retried with backoff, because in a
663    /// durable system the server coming back is the normal case.
664    pub async fn run(self, shutdown: CancellationToken) {
665        tracing::info!(
666            worker_id = %self.worker_id,
667            base_url = %self.base_url,
668            lease_ms = self.lease_ms(),
669            pools = ?self.pools,
670            worker_version = %self.worker_version.as_deref().unwrap_or(""),
671            kinds = ?self.registered_kinds(),
672            drain_grace_ms = self.drain_grace.as_millis() as u64,
673            "activity worker started"
674        );
675        let mut backoff = self.claim_backoff_base;
676        loop {
677            // The claim poll itself is raced against the token: a drain
678            // that starts mid-poll still stops this worker from taking the
679            // lease. (If the server answered just as the poll was dropped,
680            // the task is leased to a worker that never runs it — it
681            // returns at lease expiry, the same backstop as a crash.)
682            let claim = tokio::select! {
683                _ = shutdown.cancelled() => break,
684                outcome = self.claim() => outcome,
685            };
686            match claim {
687                ClaimOutcome::Task(task) => {
688                    backoff = self.claim_backoff_base;
689                    self.run_activity(*task, &shutdown).await;
690                }
691                ClaimOutcome::Empty | ClaimOutcome::Unavailable => {
692                    let delay = backoff;
693                    backoff = backoff
694                        .checked_mul(2)
695                        .unwrap_or(self.claim_backoff_max)
696                        .min(self.claim_backoff_max);
697                    tokio::select! {
698                        _ = tokio::time::sleep(delay) => {}
699                        _ = shutdown.cancelled() => {}
700                    }
701                }
702            }
703        }
704        tracing::info!(worker_id = %self.worker_id, "activity worker drained and stopped");
705    }
706
707    fn registered_kinds(&self) -> Vec<&str> {
708        let mut kinds: Vec<&str> = self.handlers.keys().map(String::as_str).collect();
709        kinds.sort_unstable();
710        kinds
711    }
712
713    fn lease_ms(&self) -> u64 {
714        u64::try_from(self.lease.as_millis()).unwrap_or(u64::MAX)
715    }
716
717    /// One claim poll against `POST /tasks/claim`.
718    async fn claim(&self) -> ClaimOutcome {
719        let body = ClaimRequest {
720            worker_id: &self.worker_id,
721            pools: if self.pools.is_empty() {
722                None
723            } else {
724                Some(self.pools.as_slice())
725            },
726            worker_version: self.worker_version.as_deref(),
727            lease_ms: self.lease_ms(),
728        };
729        let response = match self
730            .client
731            .post(format!("{}/tasks/claim", self.base_url))
732            .json(&body)
733            .send()
734            .await
735        {
736            Ok(response) => response,
737            Err(e) => {
738                tracing::warn!(error = %e, "claim request failed");
739                return ClaimOutcome::Unavailable;
740            }
741        };
742        match response.status() {
743            StatusCode::OK => match response.json::<ClaimResponse>().await {
744                Ok(claimed) => ClaimOutcome::Task(Box::new(claimed.task)),
745                Err(e) => {
746                    tracing::warn!(error = %e, "claim returned an undecodable task body");
747                    ClaimOutcome::Unavailable
748                }
749            },
750            StatusCode::NO_CONTENT => ClaimOutcome::Empty,
751            status => {
752                tracing::warn!(%status, "claim rejected by server");
753                ClaimOutcome::Unavailable
754            }
755        }
756    }
757
758    /// Execute one claimed task and settle it, inside a `rusty.activity`
759    /// span. The one case without a settle call is lease loss: the server
760    /// owns the task again, so any call from us would `409`. (Drain-grace
761    /// expiry is the deliberate second case — see
762    /// [`AttemptOutcome::DrainGraceExpired`].)
763    async fn run_activity(&self, task: ClaimedTask, shutdown: &CancellationToken) {
764        let span = tracing::info_span!(
765            "rusty.activity",
766            worker_id = %self.worker_id,
767            task_id = %task.task_id,
768            kind = %task.kind,
769            attempt = task.attempt,
770            max_attempts = task.max_attempts,
771            run_id = %task.run_id.as_deref().unwrap_or(""),
772            thread_id = %task.thread_id.as_deref().unwrap_or(""),
773        );
774        async move {
775            let Some(handler) = self.handlers.get(&task.kind).cloned() else {
776                self.fail(
777                    &task.task_id,
778                    ErrorClass::InvalidInput,
779                    format!(
780                        "no activity registered for kind `{}` on this worker (registered: {:?})",
781                        task.kind,
782                        self.registered_kinds()
783                    ),
784                    false,
785                )
786                .await;
787                return;
788            };
789
790            // Work that is already cancelled must not start: report the
791            // attempt cancelled without running the handler. (A
792            // well-behaved server never leases such a task; this is the
793            // worker-side defense of the same rule.)
794            if task.cancel_requested {
795                tracing::warn!("claimed task was already cancel-requested; reporting cancelled");
796                self.fail(
797                    &task.task_id,
798                    ErrorClass::Cancelled,
799                    format!("activity `{}` cancelled before dispatch", task.kind),
800                    false,
801                )
802                .await;
803                return;
804            }
805            // Wall-clock remaining until the whole-task deadline; a
806            // deadline already passed is reported cancelled without
807            // running the handler.
808            let until_deadline = match task.deadline {
809                Some(deadline) => match (deadline - Utc::now()).to_std() {
810                    Ok(remaining) => Some(remaining),
811                    Err(_) => {
812                        tracing::warn!(%deadline, "claimed task's deadline already passed; reporting cancelled");
813                        self.fail(
814                            &task.task_id,
815                            ErrorClass::Cancelled,
816                            format!(
817                                "activity `{}` cancelled: whole-task deadline already passed",
818                                task.kind
819                            ),
820                            false,
821                        )
822                        .await;
823                        return;
824                    }
825                },
826                None => None,
827            };
828
829            let ctx = ActivityContext {
830                task_id: task.task_id.clone(),
831                kind: task.kind.clone(),
832                attempt: task.attempt,
833                idempotency_key: task.idempotency_key.clone(),
834                payload: task.payload.clone(),
835            };
836            let task_id = task.task_id.clone();
837
838            let lease_lost = CancellationToken::new();
839            let cancel_requested = CancellationToken::new();
840            let heartbeat =
841                self.spawn_heartbeat(&task_id, lease_lost.clone(), cancel_requested.clone());
842
843            // The handler runs on its own task for the same reason as in
844            // `/execute`: a panic must surface as an outcome, not a
845            // torn-down worker.
846            let mut handle = tokio::spawn(async move { handler.run_with_receipt(ctx).await });
847            // The drain bound: starts counting when the drain starts (not
848            // at claim time), so a worker idle for hours and drained
849            // mid-attempt still gets the full grace for that attempt.
850            let drain_grace = self.drain_grace;
851            let drained = shutdown.clone();
852            let outcome = tokio::select! {
853                joined = &mut handle => AttemptOutcome::Finished(joined),
854                _ = lease_lost.cancelled() => AttemptOutcome::LeaseLost,
855                _ = cancel_requested.cancelled() => AttemptOutcome::Cancelled("cancelled by the control plane"),
856                _ = sleep_until_deadline(until_deadline) => AttemptOutcome::Cancelled("whole-task deadline expired mid-attempt"),
857                _ = async {
858                    drained.cancelled().await;
859                    tokio::time::sleep(drain_grace).await;
860                } => AttemptOutcome::DrainGraceExpired,
861            };
862            heartbeat.abort();
863
864            let joined = match outcome {
865                AttemptOutcome::Finished(joined) => joined,
866                AttemptOutcome::LeaseLost => {
867                    // Dropping the JoinHandle alone would detach the handler and
868                    // let it keep running; abort it and await the abort so the
869                    // handler future is really dropped before the next claim.
870                    handle.abort();
871                    let _ = handle.await;
872                    tracing::warn!("lease lost; activity aborted and left for the server to reassign");
873                    return;
874                }
875                AttemptOutcome::DrainGraceExpired => {
876                    // Same abort discipline as lease loss, and the same "no
877                    // settle call" — but by choice: the lease is still
878                    // held, and reporting `cancelled` would kill the task
879                    // (terminal, never retried). Leaving it unsettled
880                    // releases the task back to visibility at lease expiry
881                    // for a worker that is still serving.
882                    handle.abort();
883                    let _ = handle.await;
884                    tracing::warn!(
885                        drain_grace_ms = drain_grace.as_millis() as u64,
886                        "drain grace expired; activity aborted and left for reassignment at lease expiry"
887                    );
888                    return;
889                }
890                AttemptOutcome::Cancelled(reason) => {
891                    // Same abort discipline as lease loss, but here we still
892                    // hold the lease: settle the attempt as cancelled so the
893                    // record ends terminal-cancelled rather than waiting out
894                    // the lease.
895                    handle.abort();
896                    let _ = handle.await;
897                    tracing::warn!(reason, "activity aborted");
898                    self.fail(
899                        &task_id,
900                        ErrorClass::Cancelled,
901                        format!("activity `{}` {reason}", task.kind),
902                        false,
903                    )
904                    .await;
905                    return;
906                }
907            };
908
909            match joined {
910                Ok(Ok(completion)) => {
911                    tracing::info!(receipt = completion.receipt.is_some(), "activity succeeded");
912                    self.complete(&task_id, &completion.result, completion.receipt.as_ref())
913                        .await;
914                }
915                Ok(Err(e)) => {
916                    let (error_class, retryable) = classify_error(&e);
917                    tracing::warn!(error = %e, ?error_class, retryable, "activity failed");
918                    self.fail(
919                        &task_id,
920                        error_class,
921                        format!("activity `{}` failed: {e}", task.kind),
922                        retryable,
923                    )
924                    .await;
925                }
926                Err(join_err) => {
927                    let detail = if join_err.is_panic() {
928                        let payload = join_err.into_panic();
929                        let message = payload
930                            .downcast_ref::<&str>()
931                            .map(|s| (*s).to_owned())
932                            .or_else(|| payload.downcast_ref::<String>().cloned())
933                            .unwrap_or_else(|| "non-string panic payload".to_owned());
934                        format!("handler panicked: {message}")
935                    } else {
936                        "handler task cancelled".to_owned()
937                    };
938                    tracing::warn!(%detail, "activity panicked");
939                    self.fail(
940                        &task_id,
941                        ErrorClass::Unknown,
942                        format!("activity `{}` {detail}", task.kind),
943                        false,
944                    )
945                    .await;
946                }
947            }
948        }
949        .instrument(span)
950        .await
951    }
952
953    /// Spawn the background heartbeat loop for one claimed task: renew the
954    /// lease every `lease / 3`. The loop ends the worker's wait on the
955    /// handler in two cases: a `409` cancels `lease_lost` (the server
956    /// reassigns; the worker abandons), and a `200` carrying
957    /// `cancel_requested: true` cancels `cancel_requested` (the worker
958    /// aborts promptly and reports the attempt as cancelled through the
959    /// fail path — the whole point of the hint).
960    fn spawn_heartbeat(
961        &self,
962        task_id: &str,
963        lease_lost: CancellationToken,
964        cancel_requested: CancellationToken,
965    ) -> tokio::task::JoinHandle<()> {
966        let client = self.client.clone();
967        let url = format!("{}/tasks/{task_id}/heartbeat", self.base_url);
968        let worker_id = self.worker_id.clone();
969        let lease_ms = self.lease_ms();
970        let interval = (self.lease / 3).max(Duration::from_millis(1));
971        tokio::spawn(async move {
972            let mut ticker = tokio::time::interval(interval);
973            // A delayed tick must not burst: heartbeat spacing is a liveness
974            // signal, not a quota to catch up on.
975            ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
976            loop {
977                ticker.tick().await;
978                let body = HeartbeatRequest {
979                    worker_id: &worker_id,
980                    lease_ms,
981                };
982                match client.post(&url).json(&body).send().await {
983                    Ok(response) if response.status() == StatusCode::OK => {
984                        match response.json::<HeartbeatResponse>().await {
985                            Ok(renewed) => {
986                                if renewed.cancel_requested {
987                                    tracing::warn!(
988                                        "heartbeat carried cancel_requested; aborting the attempt"
989                                    );
990                                    cancel_requested.cancel();
991                                    return;
992                                }
993                                let expires_at = renewed
994                                    .lease_expires_at
995                                    .unwrap_or_else(|| "<missing>".to_owned());
996                                tracing::debug!(lease_expires_at = %expires_at, "lease renewed");
997                            }
998                            Err(_) => {
999                                // An undecodable renewal is not a lease-loss
1000                                // signal; the 409 is. Keep heartbeating.
1001                                tracing::warn!("heartbeat reply undecodable; will retry");
1002                            }
1003                        }
1004                    }
1005                    Ok(response) if response.status() == StatusCode::CONFLICT => {
1006                        tracing::warn!("heartbeat rejected (409): lease lost");
1007                        lease_lost.cancel();
1008                        return;
1009                    }
1010                    Ok(response) => {
1011                        // Any other status is treated as transient: the
1012                        // server will answer 409 once the lease is really
1013                        // gone, which is the authoritative signal.
1014                        tracing::warn!(status = %response.status(), "heartbeat failed; will retry");
1015                    }
1016                    Err(e) => {
1017                        tracing::warn!(error = %e, "heartbeat request failed; will retry");
1018                    }
1019                }
1020            }
1021        })
1022    }
1023
1024    /// Settle a task successfully via `POST /tasks/{id}/complete`, carrying
1025    /// the effect receipt when the activity reported one.
1026    async fn complete(&self, task_id: &str, result: &Value, receipt: Option<&EffectReceipt>) {
1027        let body = CompleteRequest {
1028            worker_id: &self.worker_id,
1029            result,
1030            receipt,
1031        };
1032        let response = self
1033            .client
1034            .post(format!("{}/tasks/{task_id}/complete", self.base_url))
1035            .json(&body)
1036            .send()
1037            .await;
1038        self.log_settle("complete", response).await;
1039    }
1040
1041    /// Settle a task as failed via `POST /tasks/{id}/fail` and log the
1042    /// server's retry decision (`{requeued, next_attempt_at, dead}`).
1043    async fn fail(&self, task_id: &str, error_class: ErrorClass, message: String, retryable: bool) {
1044        let body = FailRequest {
1045            worker_id: &self.worker_id,
1046            error_class,
1047            message: &message,
1048            retryable,
1049        };
1050        let response = self
1051            .client
1052            .post(format!("{}/tasks/{task_id}/fail", self.base_url))
1053            .json(&body)
1054            .send()
1055            .await;
1056        match response {
1057            Ok(response) if response.status() == StatusCode::OK => {
1058                match response.json::<FailResponse>().await {
1059                    Ok(outcome) => tracing::info!(
1060                        requeued = outcome.requeued,
1061                        next_attempt_at = ?outcome.next_attempt_at,
1062                        dead = outcome.dead,
1063                        "task settled as failed"
1064                    ),
1065                    Err(e) => {
1066                        tracing::warn!(error = %e, "task settled as failed; reply undecodable")
1067                    }
1068                }
1069            }
1070            other => self.log_settle("fail", other).await,
1071        }
1072    }
1073
1074    /// Shared settle-call logging: a `409` means the server already considers
1075    /// the task lost or settled (our outcome is dropped, its reassignment is
1076    /// authoritative); a transport failure means the task returns to the
1077    /// queue when the lease expires.
1078    async fn log_settle(&self, call: &str, response: reqwest::Result<reqwest::Response>) {
1079        match response {
1080            Ok(response) if response.status() == StatusCode::OK => {
1081                tracing::debug!("{call} accepted");
1082            }
1083            Ok(response) if response.status() == StatusCode::CONFLICT => {
1084                tracing::warn!(
1085                    "{call} rejected (409): task already lost or settled; \
1086                     outcome dropped in favor of the server's reassignment"
1087                );
1088            }
1089            Ok(response) => {
1090                tracing::warn!(status = %response.status(), "{call} call failed");
1091            }
1092            Err(e) => {
1093                tracing::error!(
1094                    error = %e,
1095                    "{call} call failed; the task returns to the queue at lease expiry"
1096                );
1097            }
1098        }
1099    }
1100}
1101
1102#[cfg(test)]
1103mod tests {
1104    use super::*;
1105    use serde_json::json;
1106
1107    #[test]
1108    fn closure_and_arc_implement_activity() {
1109        let closure = |ctx: ActivityContext| async move { Ok(ctx.payload().clone()) };
1110        fn assert_activity<T: Activity>(_: &T) {}
1111        assert_activity(&closure);
1112
1113        let arc: Arc<dyn Activity> = Arc::new(closure);
1114        assert_activity(&arc);
1115    }
1116
1117    #[tokio::test]
1118    async fn run_with_receipt_defaults_to_no_receipt() {
1119        // Pre-receipt activities (and the closure blanket impl) complete
1120        // through the provided method unchanged: their result is preserved
1121        // and no receipt is invented on their behalf.
1122        let closure = |ctx: ActivityContext| async move { Ok(json!({"echo": ctx.payload()})) };
1123        let ctx = ActivityContext {
1124            task_id: "t-1".into(),
1125            kind: "k".into(),
1126            attempt: 1,
1127            idempotency_key: None,
1128            payload: json!({"n": 1}),
1129        };
1130        let completion = closure.run_with_receipt(ctx).await.unwrap();
1131        assert_eq!(completion.result, json!({"echo": {"n": 1}}));
1132        assert!(completion.receipt.is_none());
1133    }
1134
1135    #[tokio::test]
1136    async fn run_with_receipt_carries_the_reported_receipt() {
1137        struct Charger;
1138        #[async_trait]
1139        impl Activity for Charger {
1140            async fn run(&self, _ctx: ActivityContext) -> Result<Value> {
1141                unreachable!("the worker drives run_with_receipt")
1142            }
1143            async fn run_with_receipt(&self, ctx: ActivityContext) -> Result<ActivityCompletion> {
1144                Ok(ActivityCompletion {
1145                    result: json!({"charged": true}),
1146                    receipt: Some(EffectReceipt {
1147                        provider: "stripe".into(),
1148                        provider_id: "ch_3PKd".into(),
1149                        idempotency_key: ctx.idempotency_key().unwrap_or_default().to_owned(),
1150                        task_id: Some(ctx.task_id().to_owned()),
1151                        // R0.7: the typed-kernel effect id is unset on this
1152                        // pre-kernel receipt path; the idempotency-key lookup
1153                        // keeps serving it.
1154                        effect_id: None,
1155                    }),
1156                })
1157            }
1158        }
1159        let ctx = ActivityContext {
1160            task_id: "t-7".into(),
1161            kind: "charge_card".into(),
1162            attempt: 1,
1163            idempotency_key: Some("run-9:charge:7".into()),
1164            payload: json!({}),
1165        };
1166        let completion = Charger.run_with_receipt(ctx).await.unwrap();
1167        let receipt = completion.receipt.expect("receipt reported");
1168        assert_eq!(receipt.provider_id, "ch_3PKd");
1169        assert_eq!(receipt.idempotency_key, "run-9:charge:7");
1170        assert_eq!(receipt.task_id.as_deref(), Some("t-7"));
1171    }
1172
1173    #[tokio::test]
1174    async fn arc_dyn_activity_delegates_run_with_receipt() {
1175        // Regression: the worker stores every registered handler as
1176        // `Arc<dyn Activity>` and drives it through `run_with_receipt`.
1177        // When the `Arc<dyn Activity>` impl overrode only `run`, the
1178        // default `run_with_receipt` answered for the Arc and silently
1179        // dropped the receipt — the crash-recovery proof caught it.
1180        struct Charger;
1181        #[async_trait]
1182        impl Activity for Charger {
1183            async fn run(&self, _ctx: ActivityContext) -> Result<Value> {
1184                unreachable!("the worker drives run_with_receipt")
1185            }
1186            async fn run_with_receipt(&self, ctx: ActivityContext) -> Result<ActivityCompletion> {
1187                Ok(ActivityCompletion {
1188                    result: json!({"charged": true}),
1189                    receipt: Some(EffectReceipt {
1190                        provider: "stripe".into(),
1191                        provider_id: "ch_arc".into(),
1192                        idempotency_key: ctx.idempotency_key().unwrap_or_default().to_owned(),
1193                        task_id: Some(ctx.task_id().to_owned()),
1194                        // R0.7: the typed-kernel effect id is unset on this
1195                        // pre-kernel receipt path; the idempotency-key lookup
1196                        // keeps serving it.
1197                        effect_id: None,
1198                    }),
1199                })
1200            }
1201        }
1202        let activity: Arc<dyn Activity> = Arc::new(Charger);
1203        let ctx = ActivityContext {
1204            task_id: "t-9".into(),
1205            kind: "charge_card".into(),
1206            attempt: 2,
1207            idempotency_key: Some("run-1:charge:9".into()),
1208            payload: json!({}),
1209        };
1210        let completion = activity.run_with_receipt(ctx).await.unwrap();
1211        assert_eq!(completion.result, json!({"charged": true}));
1212        let receipt = completion
1213            .receipt
1214            .expect("the Arc must delegate the receipt");
1215        assert_eq!(receipt.provider_id, "ch_arc");
1216        assert_eq!(receipt.idempotency_key, "run-1:charge:9");
1217    }
1218
1219    #[test]
1220    fn classify_error_maps_to_the_shared_taxonomy() {
1221        // The transient executor classes are upstream dependency failures.
1222        assert_eq!(
1223            classify_error(&RustyError::Llm("rate limited".into())),
1224            (ErrorClass::DependencyFailure, true)
1225        );
1226        assert_eq!(
1227            classify_error(&RustyError::Tool("backend exploded".into())),
1228            (ErrorClass::DependencyFailure, true)
1229        );
1230        // Contract violations are invalid input.
1231        assert_eq!(
1232            classify_error(&RustyError::Graph("cycle".into())),
1233            (ErrorClass::InvalidInput, false)
1234        );
1235        assert_eq!(
1236            classify_error(&RustyError::InvalidUpdate("double write".into())),
1237            (ErrorClass::InvalidInput, false)
1238        );
1239        // Everything else is unclassified and hard.
1240        assert_eq!(
1241            classify_error(&RustyError::Node("boom".into())),
1242            (ErrorClass::Unknown, false)
1243        );
1244        assert_eq!(
1245            classify_error(&RustyError::Checkpoint("io".into())),
1246            (ErrorClass::Unknown, false)
1247        );
1248        assert_eq!(
1249            classify_error(&RustyError::Replay("diverged".into())),
1250            (ErrorClass::Unknown, false)
1251        );
1252        let serde_err = serde_json::from_str::<Value>("{").unwrap_err();
1253        assert_eq!(
1254            classify_error(&RustyError::Serialization(serde_err)),
1255            (ErrorClass::Unknown, false)
1256        );
1257        // Interrupts have no suspend semantics here; they settle as a
1258        // non-retryable cancellation.
1259        assert_eq!(
1260            classify_error(&RustyError::Interrupt { value: json!(null) }),
1261            (ErrorClass::Cancelled, false)
1262        );
1263    }
1264
1265    #[test]
1266    fn claimed_task_decodes_the_server_wire_shape() {
1267        // The claim response wraps the task record: `{"task": {…}}`. Fields
1268        // the worker does not model (pool, status, lease, timestamps) must
1269        // be tolerated.
1270        let body = json!({
1271            "task": {
1272                "task_id": "t-1",
1273                "kind": "send_email",
1274                "payload": {"to": "a@b.c"},
1275                "pool": "default",
1276                "status": "leased",
1277                "attempt": 2,
1278                "max_attempts": 3,
1279                "error_class": null,
1280                "last_error": "previous attempt blew up",
1281                "idempotency_key": "run-9:charge:7",
1282                "result": null,
1283                "run_id": "run-9",
1284                "thread_id": "thread-1",
1285                "cancel_requested": false,
1286                "deadline": "2026-08-07T13:00:00Z",
1287                "lease": {"owner": "w-1", "expires_at": "2026-08-07T12:00:30Z"},
1288                "next_attempt_at": null,
1289                "created_at": "2026-08-07T12:00:00Z",
1290                "updated_at": "2026-08-07T12:00:01Z"
1291            }
1292        });
1293        let claimed: ClaimResponse = serde_json::from_value(body).unwrap();
1294        let task = claimed.task;
1295        assert_eq!(task.task_id, "t-1");
1296        assert_eq!(task.kind, "send_email");
1297        assert_eq!(task.payload, json!({"to": "a@b.c"}));
1298        assert_eq!(task.attempt, 2);
1299        assert_eq!(task.max_attempts, 3);
1300        assert_eq!(task.idempotency_key.as_deref(), Some("run-9:charge:7"));
1301        assert_eq!(task.run_id.as_deref(), Some("run-9"));
1302        assert_eq!(task.thread_id.as_deref(), Some("thread-1"));
1303        assert!(!task.cancel_requested);
1304        assert_eq!(
1305            task.deadline,
1306            DateTime::parse_from_rfc3339("2026-08-07T13:00:00Z")
1307                .ok()
1308                .map(|dt| dt.with_timezone(&Utc))
1309        );
1310    }
1311
1312    #[test]
1313    fn claimed_task_defaults_optional_fields() {
1314        // A pre-wave-2 server omits the cancellation fields entirely.
1315        let task: ClaimedTask =
1316            serde_json::from_value(json!({"task_id": "t-1", "kind": "k"})).unwrap();
1317        assert_eq!(task.payload, Value::Null);
1318        assert_eq!(task.attempt, 0);
1319        assert!(task.idempotency_key.is_none());
1320        assert!(task.run_id.is_none() && task.thread_id.is_none());
1321        assert!(!task.cancel_requested);
1322        assert!(task.deadline.is_none());
1323    }
1324
1325    #[test]
1326    fn claim_request_omits_pools_when_unconfigured() {
1327        let body = ClaimRequest {
1328            worker_id: "w-1",
1329            pools: None,
1330            worker_version: None,
1331            lease_ms: 100,
1332        };
1333        let value = serde_json::to_value(&body).unwrap();
1334        assert_eq!(value, json!({"worker_id": "w-1", "lease_ms": 100}));
1335
1336        let pools = vec!["gpu".to_string()];
1337        let body = ClaimRequest {
1338            worker_id: "w-1",
1339            pools: Some(pools.as_slice()),
1340            worker_version: None,
1341            lease_ms: 100,
1342        };
1343        let value = serde_json::to_value(&body).unwrap();
1344        assert_eq!(
1345            value,
1346            json!({"worker_id": "w-1", "pools": ["gpu"], "lease_ms": 100})
1347        );
1348    }
1349
1350    #[test]
1351    fn claim_request_carries_the_advertised_worker_version() {
1352        // Wave 3: a configured version travels on the claim; unconfigured
1353        // the field is omitted entirely (pre-wave-3 servers never see it).
1354        let body = ClaimRequest {
1355            worker_id: "w-1",
1356            pools: None,
1357            worker_version: Some("activity-worker/1.4.0"),
1358            lease_ms: 100,
1359        };
1360        let value = serde_json::to_value(&body).unwrap();
1361        assert_eq!(
1362            value,
1363            json!({"worker_id": "w-1", "worker_version": "activity-worker/1.4.0", "lease_ms": 100})
1364        );
1365    }
1366
1367    #[test]
1368    fn settle_request_shapes_match_the_contract() {
1369        let heartbeat = HeartbeatRequest {
1370            worker_id: "w-1",
1371            lease_ms: 100,
1372        };
1373        assert_eq!(
1374            serde_json::to_value(&heartbeat).unwrap(),
1375            json!({"worker_id": "w-1", "lease_ms": 100})
1376        );
1377
1378        let result = json!({"sent": true});
1379        let complete = CompleteRequest {
1380            worker_id: "w-1",
1381            result: &result,
1382            receipt: None,
1383        };
1384        assert_eq!(
1385            serde_json::to_value(&complete).unwrap(),
1386            json!({"worker_id": "w-1", "result": {"sent": true}})
1387        );
1388
1389        // An activity reporting a receipt carries it with the completion;
1390        // without one the body is byte-identical to the pre-receipt shape.
1391        let receipt = EffectReceipt {
1392            provider: "stripe".into(),
1393            provider_id: "ch_3PKd".into(),
1394            idempotency_key: "run-9:charge:7".into(),
1395            task_id: Some("t-1".into()),
1396            effect_id: None,
1397        };
1398        let complete = CompleteRequest {
1399            worker_id: "w-1",
1400            result: &result,
1401            receipt: Some(&receipt),
1402        };
1403        assert_eq!(
1404            serde_json::to_value(&complete).unwrap(),
1405            json!({
1406                "worker_id": "w-1",
1407                "result": {"sent": true},
1408                "receipt": {
1409                    "provider": "stripe",
1410                    "provider_id": "ch_3PKd",
1411                    "idempotency_key": "run-9:charge:7",
1412                    "task_id": "t-1"
1413                }
1414            })
1415        );
1416
1417        let fail = FailRequest {
1418            worker_id: "w-1",
1419            error_class: ErrorClass::DependencyFailure,
1420            message: "backend exploded",
1421            retryable: true,
1422        };
1423        assert_eq!(
1424            serde_json::to_value(&fail).unwrap(),
1425            json!({
1426                "worker_id": "w-1",
1427                "error_class": "dependency_failure",
1428                "message": "backend exploded",
1429                "retryable": true
1430            })
1431        );
1432    }
1433
1434    #[test]
1435    fn heartbeat_and_fail_responses_decode() {
1436        let heartbeat: HeartbeatResponse =
1437            serde_json::from_value(json!({"lease_expires_at": "2026-08-07T12:00:30Z"})).unwrap();
1438        assert_eq!(
1439            heartbeat.lease_expires_at.as_deref(),
1440            Some("2026-08-07T12:00:30Z")
1441        );
1442        // The cancellation hint defaults off (pre-wave-2 servers omit it).
1443        assert!(!heartbeat.cancel_requested);
1444
1445        let heartbeat: HeartbeatResponse = serde_json::from_value(
1446            json!({"lease_expires_at": "2026-08-07T12:00:30Z", "cancel_requested": true}),
1447        )
1448        .unwrap();
1449        assert!(heartbeat.cancel_requested);
1450
1451        let fail: FailResponse = serde_json::from_value(json!({
1452            "requeued": true,
1453            "next_attempt_at": "2026-08-07T12:01:00Z",
1454            "dead": false
1455        }))
1456        .unwrap();
1457        assert!(fail.requeued && !fail.dead);
1458        assert_eq!(
1459            fail.next_attempt_at.as_deref(),
1460            Some("2026-08-07T12:01:00Z")
1461        );
1462    }
1463
1464    #[test]
1465    fn lease_is_clamped_to_the_server_bounds() {
1466        let worker = ActivityWorker::new("http://127.0.0.1:1").with_lease(Duration::from_millis(5));
1467        assert_eq!(worker.lease, MIN_LEASE);
1468        let worker =
1469            ActivityWorker::new("http://127.0.0.1:1").with_lease(Duration::from_secs(86_400));
1470        assert_eq!(worker.lease, MAX_LEASE);
1471    }
1472}