Skip to main content

aion_core/
cluster_event.rs

1//! Cluster topology and ownership events for the ops console real-time channel (WS3).
2//!
3//! This module defines the *typed contract* for the second class of real-time push the ops console
4//! consumes: cluster/topology/ownership deltas that today exist only as `tracing` logs and
5//! Prometheus gauges. The wire shapes live in `aion-core` (not `aion-server`) for one reason: only
6//! this leaf crate depends on `ts-rs`, so this is the single place a Rust type can cross the
7//! Rust -> TypeScript boundary into the ops console's generated bindings.
8//!
9//! # Scope (WS3 FOUNDATION)
10//!
11//! This file is *types only*. It deliberately does **not** define the broadcast publisher, the
12//! emit call-sites in the supervisor / registry, the subscription endpoint, or the namespace gate.
13//! Those are wired in later WS3 increments. Defining the contract first lets the ops console and the
14//! server agree on shapes before any behaviour ships.
15//!
16//! # Honesty corrections applied to the original design
17//!
18//! The design table named several variants whose payloads cannot be sourced from the subsystems
19//! that would emit them today. Rather than ship a type whose fields are guaranteed to be faked,
20//! those variants are **descoped from Phase 1** and documented here so the omission is explicit and
21//! not silently rediscovered at wiring time:
22//!
23//! - **`NodeMetricsSampled` — DEFERRED.** `aion-server`'s metrics (`observability/metrics.rs`) have
24//!   no `node` dimension (`connected_workers` is an `IntGaugeVec` keyed by *namespace*) and there
25//!   is no `workflows_running` gauge at all. A `{ node, connected_workers, workflows_running }`
26//!   payload cannot be produced by "piggybacking the gauge setters". Sourcing it honestly requires
27//!   a real metrics change (node-labelled gauges + a running gauge). Until then the ops console
28//!   derives `connected_workers` client-side from [`ClusterEvent::WorkerConnected`] /
29//!   [`ClusterEvent::WorkerDisconnected`] deltas against the [`ClusterSnapshot`] baseline. Adding
30//!   a timer that scrapes Prometheus to fill this variant is the exact polling-as-push regression
31//!   WS3 exists to remove, so the variant is omitted rather than tempting that shortcut.
32//! - **`ShardOwnerChanged` / `FencedCasRejected` — DEFERRED (fast-follow).** The haematite seam
33//!   (`HaematiteStore::publish_ln(shard)`) carries only `shard`; the fenced-CAS reject is a
34//!   `haematite::DatabaseError::Fenced { .. }` destructured with `..` and mapped to
35//!   `StoreError::NotOwner { shard }`, carrying only `shard` — never the owner name or the
36//!   attempted/current epochs. The proposed payloads require enriching haematite's `Fenced` error
37//!   to surface owner identity and epoch, which is a cross-crate (likely cross-repo) change. The
38//!   cluster map is still *honest* without these two: shard adoption is fully observable from the
39//!   supervisor's `tick()` ([`ClusterEvent::ShardAdopted`] and friends). Only the store-side
40//!   CAS-reject detail is delayed.
41//!
42//! # u64 precision across the TS boundary
43//!
44//! The ts-rs config exports every `u64` as TS `number` (`with_large_int("number")` in
45//! `generated_types.rs`), which truncates above `2^53`. [`ClusterEventMeta::cluster_seq`] and the
46//! epoch fields below are `u64`. This is the *same* accepted ceiling that already applies to
47//! [`crate::EventEnvelope::seq`]; cluster sequencing follows the established project convention
48//! rather than introducing a divergent string encoding. The ceiling is documented on each field so
49//! the gap-detection math on the client is aware of the bound. A long-lived deployment must keep
50//! `cluster_seq` below `2^53`; in practice this is never reached for a topology-event counter.
51
52use chrono::{DateTime, Utc};
53use serde::{Deserialize, Serialize};
54
55use crate::{ActivityId, WorkflowId};
56
57/// Metadata stamped on every [`ClusterEvent`], mirroring [`crate::EventEnvelope`] for the cluster
58/// channel.
59#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
60pub struct ClusterEventMeta {
61    /// Monotonic sequence number assigned by the cluster publisher (a single deployment-global
62    /// `AtomicU64`). The client uses this for gap detection and `after_seq` reconnect dedup, the
63    /// cluster analog of [`crate::EventEnvelope::seq`].
64    ///
65    /// Exported to TypeScript as `number`; see the module docs for the accepted `2^53` ceiling.
66    pub cluster_seq: u64,
67    /// UTC wall-clock instant at which the originating subsystem observed the state change.
68    pub observed_at: DateTime<Utc>,
69}
70
71/// Transport a connected worker is delivered to over.
72///
73/// Mirrors the live `aion_server::worker::registry::WorkerDelivery` discriminants without carrying
74/// the (non-serializable) delivery channels, so it can cross the wire and the ts-rs boundary.
75#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
76#[serde(tag = "transport")]
77pub enum WorkerTransport {
78    /// gRPC bidirectional-stream delivery (the default transport).
79    Grpc,
80    /// Liminal server-push delivery (feature-gated on the server).
81    Liminal,
82}
83
84impl WorkerTransport {
85    /// The word an operator sees for this transport in a log line.
86    ///
87    /// Lives beside the discriminants rather than in the server, so the name a
88    /// liveness withdrawal prints and the name a cluster event carries cannot
89    /// disagree about which link an operator should go and look at.
90    #[must_use]
91    pub const fn name(self) -> &'static str {
92        match self {
93            Self::Grpc => "grpc",
94            Self::Liminal => "liminal",
95        }
96    }
97}
98
99/// Why a worker left the connected set.
100///
101/// NOTE (wiring honesty): the single registry deregistration site
102/// (`ConnectedWorkerRegistry::deregister` / `remove_worker`) does not today distinguish a transport
103/// disconnect from an idle timeout from an explicit deregister. The emit increment MUST derive the
104/// reason at the call site from real signal (e.g. a closed delivery channel vs an explicit
105/// deregister RPC vs a liveness-timeout sweep) or collapse to the variant it can actually prove.
106/// This enum defines the *contract*; it must not be populated with a fabricated distinction.
107#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
108#[serde(tag = "reason")]
109pub enum WorkerDeathReason {
110    /// The worker's delivery transport dropped (stream/connection closed).
111    Disconnect,
112    /// The worker was removed by a liveness/idle-timeout sweep.
113    Timeout,
114    /// The worker explicitly deregistered.
115    Deregistered,
116}
117
118/// Operator-requested lifecycle state for a durable worker deployment.
119#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
120#[serde(rename_all = "kebab-case")]
121pub enum DesiredState {
122    /// The deployment should run.
123    Running,
124    /// The deployment should remain stopped.
125    Stopped,
126}
127
128impl DesiredState {
129    /// Stable human-readable state token.
130    #[must_use]
131    pub const fn token(self) -> &'static str {
132        match self {
133            Self::Running => "running",
134            Self::Stopped => "stopped",
135        }
136    }
137}
138
139/// Create-or-replace result exposed on the cluster and HTTP wires.
140#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
141#[serde(rename_all = "kebab-case")]
142pub enum PutOutcome {
143    /// No record previously existed for the key.
144    Created,
145    /// A record already existed for the key and was replaced.
146    Replaced,
147}
148
149/// Deployment-record lookup state captured when a worker registers.
150#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
151#[serde(rename_all = "kebab-case")]
152pub enum DeploymentAssociation {
153    /// The named deployment record existed.
154    Known,
155    /// The deployment store was checked and the named record was absent.
156    Absent,
157    /// No deployment store was attached, so existence was not checked.
158    Unchecked,
159}
160
161/// A cluster/topology/ownership delta pushed over the ops console real-time channel.
162///
163/// Tagged union on `type` to match the existing [`crate::Event`] wire shape. Every variant carries
164/// [`ClusterEventMeta`] as `meta`. Cluster events are deployment-scoped, not namespace-stamped at
165/// the envelope level; the `Worker*` variants carry their own `namespaces` so the server-side gate
166/// can intersect them against the caller's grants (see the deferred `cluster_filter`).
167#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
168#[serde(tag = "type")]
169pub enum ClusterEvent {
170    /// A peer was added to this node's watch set (cluster-membership grew mid-session).
171    ///
172    /// Distinct from [`Self::PeerConnected`]: this is a *topology* change (a new peer to watch),
173    /// not a *liveness* transition. Without it, a peer that joins after the priming
174    /// [`ClusterSnapshot`] would silently never appear on the map until the next snapshot.
175    PeerAdded {
176        /// Cluster-event metadata.
177        meta: ClusterEventMeta,
178        /// The newly-watched peer's node name.
179        peer_name: String,
180        /// The peer's forwarding address, when known.
181        forward_addr: Option<String>,
182    },
183    /// A watched peer transitioned from down/unknown to connected (a liveness recovery).
184    ///
185    /// EMIT NOTE: the supervisor `tick()` resets `consecutive_down`/`adopted` on the same tick it
186    /// observes `connected`, so the recovery condition (`was_down = consecutive_down > 0 ||
187    /// adopted`) MUST be captured *before* that reset and the emit driven by the captured value.
188    /// Emitting after the reset produces no recovery events (every tick looks freshly connected).
189    PeerConnected {
190        /// Cluster-event metadata.
191        meta: ClusterEventMeta,
192        /// The recovered peer's node name.
193        peer_name: String,
194        /// The peer's forwarding address, when known.
195        forward_addr: Option<String>,
196    },
197    /// A watched peer was observed down on a supervisor tick.
198    ///
199    /// Emitted on every tick the peer is observed down; `confirmed` flips to `true` once
200    /// `consecutive_down >= confirmations` (the debounce threshold that authorizes adoption).
201    PeerDisconnected {
202        /// Cluster-event metadata.
203        meta: ClusterEventMeta,
204        /// The down peer's node name.
205        peer_name: String,
206        /// Consecutive ticks this peer has been observed down.
207        consecutive_down: u32,
208        /// Whether the debounce threshold has been crossed (adoption-eligible).
209        confirmed: bool,
210    },
211    /// This node adopted shards previously owned by a failed peer.
212    ShardAdopted {
213        /// Cluster-event metadata.
214        meta: ClusterEventMeta,
215        /// Shard indices adopted in this transition.
216        shards: Vec<usize>,
217        /// The peer the shards were adopted from.
218        from_peer: String,
219        /// This node's name (the new owner).
220        adopted_by: String,
221    },
222    /// An adoption attempt failed and will be retried on a subsequent tick.
223    ///
224    /// HONESTY: the original design hardcoded `will_retry: true`. That is a silent lie — a peer
225    /// later observed connected resets `adopted` and stops retrying, and the
226    /// "handled elsewhere" path terminally stops. The retry decision is not expressible as a
227    /// constant, so the field is omitted; the ops console infers retry by observing whether a
228    /// subsequent [`Self::ShardAdopted`] / [`Self::ShardAdoptionSkipped`] for the same
229    /// `(from_peer, shards)` arrives, rather than trusting a promise that may never be kept
230    /// (ADR-016 no-silent-failure).
231    ShardAdoptionFailed {
232        /// Cluster-event metadata.
233        meta: ClusterEventMeta,
234        /// Shard indices the failed attempt targeted.
235        shards: Vec<usize>,
236        /// The peer the shards would have been adopted from.
237        from_peer: String,
238        /// Human-readable adoption error.
239        error: String,
240    },
241    /// Adoption was skipped because the shards are already held by a live third-party owner.
242    ShardAdoptionSkipped {
243        /// Cluster-event metadata.
244        meta: ClusterEventMeta,
245        /// Shard indices that were skipped.
246        shards: Vec<usize>,
247        /// The peer the shards would have been adopted from.
248        from_peer: String,
249        /// The live node currently holding the shards.
250        held_by: String,
251    },
252    /// A worker joined the connected set.
253    WorkerConnected {
254        /// Cluster-event metadata.
255        meta: ClusterEventMeta,
256        /// Stable worker identifier.
257        worker_id: String,
258        /// Namespaces this worker serves.
259        namespaces: Vec<String>,
260        /// Task-queue pool this worker serves within its namespaces.
261        task_queue: String,
262        /// Delivery transport for this worker.
263        transport: WorkerTransport,
264        /// Locality/node label, when the worker reported one.
265        node: Option<String>,
266        /// Deployment named by the worker's instance identity, when supplied.
267        deployment: Option<String>,
268        /// Lookup state for the named deployment. `None` means the worker
269        /// supplied no instance identity.
270        deployment_association: Option<DeploymentAssociation>,
271    },
272    /// A worker left the connected set.
273    WorkerDisconnected {
274        /// Cluster-event metadata.
275        meta: ClusterEventMeta,
276        /// Stable worker identifier.
277        worker_id: String,
278        /// Namespaces this worker served.
279        namespaces: Vec<String>,
280        /// Why the worker left (see [`WorkerDeathReason`] wiring note).
281        reason: WorkerDeathReason,
282    },
283    /// A dispatch parked on an unserved queue with **no availability
284    /// deadline** — the wait that can hold a run forever.
285    ///
286    /// Every other selection outcome has a terminal echo the operator can see:
287    /// a served dispatch completes, and a deadline-bounded park expires into a
288    /// typed `WorkerUnavailable` refusal that reaches the workflow's own
289    /// history. The unbounded park had only a WARN line and the queryable
290    /// unserved state — nothing pushed. This variant is that push (#266 T4):
291    /// the #266 boot-ordering defect stranded recovered runs in exactly this
292    /// state, and the only sign was a log nobody was reading. The status-truth
293    /// front (#264) consumes it.
294    ///
295    /// Emitted under the same transition rule as the WARN it accompanies: once
296    /// when a park episode is first classified, and again only when the
297    /// classified reason changes. Re-observations of the same reason are
298    /// silent, so a queue nobody serves does not flood the channel.
299    ///
300    /// `reason` and `policy` carry the queue-service taxonomy's canonical
301    /// spellings (e.g. `NO_LIVE_POLLERS`, `durable_pending`) as strings rather
302    /// than server-crate enums, the same leaf-crate discipline
303    /// [`Self::NamespaceCreated`] applies to `origin`.
304    DispatchParked {
305        /// Cluster-event metadata.
306        meta: ClusterEventMeta,
307        /// Namespace the dispatch belongs to.
308        namespace: String,
309        /// Task queue the dispatch is addressed to.
310        task_queue: String,
311        /// Activity type the dispatch needs served.
312        activity_type: String,
313        /// Node label the dispatch is pinned to, if any.
314        node: Option<String>,
315        /// Canonical queue-service reason spelling (`NO_QUEUE_DECLARATION` /
316        /// `NO_LIVE_POLLERS` / `POLLERS_INCOMPATIBLE` / `POLLERS_UNREACHABLE` /
317        /// `SATURATED`).
318        reason: String,
319        /// Canonical queue-service policy spelling (`strict` /
320        /// `durable_pending`).
321        policy: String,
322        /// Workflow whose activity is parked.
323        workflow_id: WorkflowId,
324        /// Parked activity's history ordinal.
325        activity_id: ActivityId,
326        /// How long the dispatch had been waiting when this was emitted.
327        ///
328        /// Exported to TypeScript as `number`; see the module docs for the
329        /// accepted `2^53` ceiling.
330        waited_ms: u64,
331        /// Workers connected for `(namespace, task_queue)`, whatever they
332        /// serve, at the census behind this verdict.
333        workers_in_pool: usize,
334        /// Of those, workers advertising this activity type.
335        workers_serving_activity: usize,
336        /// Of those, workers that also satisfy the dispatch's node pin.
337        compatible_workers: usize,
338        /// How long ago a compatible worker was last in service, or `None`
339        /// when none has ever been seen for this address in this server's
340        /// life.
341        ///
342        /// Exported to TypeScript as `number`; see the module docs for the
343        /// accepted `2^53` ceiling.
344        last_compatible_poller_age_ms: Option<u64>,
345    },
346    /// The cluster supervisor started on this node (lifecycle).
347    ///
348    /// Lets the calm-state view (ADR-019) distinguish "supervisor running, all peers healthy" from
349    /// "supervisor not running".
350    SupervisorStarted {
351        /// Cluster-event metadata.
352        meta: ClusterEventMeta,
353        /// This node's name.
354        node: String,
355    },
356    /// The cluster supervisor stopped on this node (clean drain / shutdown).
357    SupervisorStopped {
358        /// Cluster-event metadata.
359        meta: ClusterEventMeta,
360        /// This node's name.
361        node: String,
362    },
363    /// A brand-new namespace was minted in the durable registry (Control-Plane
364    /// Phase 1, S8).
365    ///
366    /// Emitted exactly once per genuinely-new namespace at the registry's single
367    /// `MintOutcome::Created` choke-point — so the worker-register seam (S5), the
368    /// workflow-start safety net (S6), and the explicit `POST /namespaces` path
369    /// (S7) all surface the same delta. An idempotent re-reference of an existing
370    /// namespace (`MintOutcome::AlreadyExisted`) never emits this, so the ops
371    /// console appends each namespace exactly once with no refresh.
372    ///
373    /// `origin` is the stable `snake_case` label (`worker_mint` / `start_mint` /
374    /// `explicit` / `inferred_from_state`) rather than the `aion-store`
375    /// `NamespaceOrigin` enum, because this leaf crate must not depend on the
376    /// store crate; the label is the same string the audit `tracing` event logs.
377    /// `created_at` is the durable record's first-mint instant, carried so the
378    /// console's created column matches the registry without a follow-up fetch.
379    NamespaceCreated {
380        /// Cluster-event metadata.
381        meta: ClusterEventMeta,
382        /// The minted namespace's name (registry primary key).
383        name: String,
384        /// The durable record's creation instant (its `created_at`).
385        created_at: DateTime<Utc>,
386        /// How the namespace came to exist, as the stable `snake_case` label.
387        origin: String,
388    },
389    /// A namespace's durable placement directive was changed (Control-Plane
390    /// Phase 2, P2-P2 — `PUT /namespaces/{name}/placement`).
391    ///
392    /// Emitted on the SAME deploy-scoped channel as [`Self::NamespaceCreated`]
393    /// after the placement is durably set, so the ops console's namespace panel
394    /// reflects an operator's placement change live with no refresh. `placement`
395    /// is the stable wire projection ([`NamespacePlacementWire`]) of the durable
396    /// `NamespacePlacement`, carried as a `kind` label + label set rather than the
397    /// `aion-store` enum, because this leaf crate must not depend on the store
398    /// crate (the same discipline [`Self::NamespaceCreated`] applies to `origin`).
399    NamespacePlacementChanged {
400        /// Cluster-event metadata.
401        meta: ClusterEventMeta,
402        /// The namespace whose placement changed (registry primary key).
403        name: String,
404        /// The new placement directive, as the stable wire projection.
405        placement: NamespacePlacementWire,
406    },
407    /// A periodic snapshot of a namespace's concurrency-quota state (Control-Plane
408    /// Phase 2, P2-Q3), pushed on the SAME deploy-scoped channel as
409    /// [`Self::NamespaceCreated`] so the ops console renders a live "in-flight /
410    /// ceiling" badge that ticks as work flows.
411    ///
412    /// This is the ONE edge-triggered exception's honest counterweight: unlike the
413    /// other variants (each emitted at a real subsystem mutation), quota state
414    /// changes on every claim/settle, so per-row emission would be a firehose.
415    /// Instead a throttled snapshot task samples each active namespace's REAL
416    /// durable state once per cadence and emits this — a periodic snapshot per
417    /// active namespace, never a client poll. The console folds the latest snapshot
418    /// per namespace, superseding the prior value.
419    ///
420    /// `in_flight` is the durable **Claimed** outbox-row count for the namespace
421    /// (`count_claimed_outbox_rows`) — the SAME notion the keyed backpressure caps,
422    /// never the dead `inflight_activities` gauge (P2-Q0). `ceiling` is the tenant's
423    /// **cluster-wide** contract: its explicit `max_in_flight_activities` override
424    /// or the platform default — the number the operator set, not the per-node
425    /// proportional slice (exposing per-node math is the leaky-abstraction footgun
426    /// §3.6 rejects). On a single node the durable `in_flight` this node sees is the
427    /// whole cluster's; multi-node exact aggregation is the reserved §3.6 follow-up.
428    NamespaceQuotaState {
429        /// Cluster-event metadata.
430        meta: ClusterEventMeta,
431        /// The namespace this quota snapshot describes (registry primary key).
432        namespace: String,
433        /// Durable count of currently-**Claimed** outbox rows for the namespace —
434        /// the in-flight activities the ceiling caps.
435        ///
436        /// Exported to TypeScript as `number`; see the module docs for the accepted
437        /// `2^53` ceiling (an in-flight count never approaches it).
438        in_flight: u64,
439        /// The tenant's cluster-wide concurrency ceiling: its explicit
440        /// `max_in_flight_activities` override, or the platform default when unset.
441        ceiling: u32,
442    },
443    /// A worker deployment was durably created or replaced by an operator.
444    WorkerDeploymentPut {
445        /// Cluster-event metadata.
446        meta: ClusterEventMeta,
447        /// Durable deployment name.
448        name: String,
449        /// Typed create-or-replace outcome.
450        outcome: PutOutcome,
451        /// Typed desired state.
452        desired_state: DesiredState,
453        /// Version captured from the deployed server binary.
454        binary_version: String,
455        /// SHA-256 captured from the deployed server executable bytes.
456        binary_content_hash: String,
457    },
458    /// An operator durably changed a deployment's desired state.
459    WorkerDeploymentDesiredStateChanged {
460        /// Cluster-event metadata.
461        meta: ClusterEventMeta,
462        /// Durable deployment name.
463        name: String,
464        /// Typed desired state.
465        desired_state: DesiredState,
466    },
467    /// An operator durably deleted a deployment record.
468    WorkerDeploymentDeleted {
469        /// Cluster-event metadata.
470        meta: ClusterEventMeta,
471        /// Deleted deployment name.
472        name: String,
473    },
474}
475
476/// Stable wire projection of a namespace's durable placement directive
477/// (`NamespacePlacement` in `aion-store`), for the ops-console real-time channel
478/// (Control-Plane Phase 2).
479///
480/// Lives in this leaf crate (not `aion-server` / `aion-store`) for the same reason
481/// the rest of [`ClusterEvent`] does: only this crate crosses the Rust ->
482/// TypeScript boundary via `ts-rs`. `kind` is the stable `snake_case` variant tag
483/// (`unplaced` / `prefer` / `pinned`) and `nodes` is the (possibly empty)
484/// node-label set; `Unplaced` carries an empty `nodes`. Modelling it as a flat
485/// `{kind, nodes}` shape rather than re-encoding the store's tagged form keeps the
486/// generated TS binding a single simple type the console can switch on.
487#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
488pub struct NamespacePlacementWire {
489    /// Stable `snake_case` placement-kind tag: `unplaced` / `prefer` / `pinned`.
490    pub kind: String,
491    /// The placement's node-label set, deterministically ordered. Empty for
492    /// `unplaced`; the preferred/required labels for `prefer`/`pinned`.
493    pub nodes: Vec<String>,
494}
495
496/// A peer entry in the priming [`ClusterSnapshot`].
497#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
498pub struct ClusterPeer {
499    /// The peer's node name.
500    pub peer_name: String,
501    /// The peer's forwarding address, when known.
502    pub forward_addr: Option<String>,
503    /// Whether the peer is currently observed connected.
504    pub connected: bool,
505    /// Consecutive ticks observed down (0 when connected).
506    pub consecutive_down: u32,
507}
508
509/// A shard ownership entry in the priming [`ClusterSnapshot`].
510#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
511pub struct ClusterShard {
512    /// Shard index.
513    pub shard: usize,
514    /// The node that currently owns the shard.
515    pub owner: String,
516    /// The epoch fence value at which the owner holds the shard.
517    ///
518    /// Exported to TypeScript as `number`; see the module docs for the accepted `2^53` ceiling.
519    pub epoch: u64,
520}
521
522/// A connected-worker entry in the priming [`ClusterSnapshot`].
523#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
524pub struct ClusterWorker {
525    /// Stable worker identifier.
526    pub worker_id: String,
527    /// Namespaces this worker serves (already intersected with the caller's grants by the gate).
528    pub namespaces: Vec<String>,
529    /// Task-queue pool this worker serves.
530    pub task_queue: String,
531    /// Delivery transport for this worker.
532    pub transport: WorkerTransport,
533    /// Locality/node label, when reported.
534    pub node: Option<String>,
535    /// Deployment named by the worker's instance identity, when supplied.
536    pub deployment: Option<String>,
537    /// Lookup state for that deployment. `None` for workers without instance
538    /// identity.
539    pub deployment_association: Option<DeploymentAssociation>,
540}
541
542/// A durable worker-deployment entry in the priming cluster snapshot.
543#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
544pub struct ClusterDeployment {
545    /// Durable deployment name.
546    pub name: String,
547    /// Typed desired state.
548    pub desired_state: DesiredState,
549    /// Version captured at deploy time.
550    pub binary_version: String,
551    /// SHA-256 captured at deploy time.
552    pub binary_content_hash: String,
553}
554
555/// A calm-state baseline of the whole cluster, sent as the priming reply before the live delta
556/// stream so the ops console can render an at-a-glance "all clear" before any [`ClusterEvent`]
557/// arrives (ADR-019). On `cluster_lagged` the client re-requests this rather than replaying a
558/// (non-durable) delta history.
559#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
560pub struct ClusterSnapshot {
561    /// The reading node's own name (self-identity; baselines the `Peer*` deltas which describe
562    /// *other* nodes).
563    pub node: String,
564    /// The `cluster_seq` this snapshot is consistent as-of; the client applies only deltas with a
565    /// strictly greater `cluster_seq`.
566    ///
567    /// Exported to TypeScript as `number`; see the module docs for the accepted `2^53` ceiling.
568    pub as_of_seq: u64,
569    /// Watched peers and their current liveness.
570    pub peers: Vec<ClusterPeer>,
571    /// Shards owned by (or visible to) the reading node.
572    pub shards: Vec<ClusterShard>,
573    /// Connected workers, already gated to the caller's namespaces.
574    pub workers: Vec<ClusterWorker>,
575    /// Durable worker deployments visible under the deploy grant.
576    pub deployments: Vec<ClusterDeployment>,
577}
578
579/// A command the ops console can issue against the cluster channel (ADR-020 command seam).
580///
581/// Defined now for contract coherence. **Phase 1 ships only [`Self::RequestClusterSnapshot`]** (a
582/// read). The mutating variants compile so the contract exists, but their handlers reject with an
583/// `unimplemented` wire error — and, per ADR-020, MUST still run the full auth gate
584/// (`caller.deploy_granted()`) *before* rejecting, so the seam's authorization contract is
585/// exercised now and an `unimplemented` stub is never an auth-bypass-shaped hole.
586///
587/// Tagged on `command` (distinct from [`ClusterEvent`]'s `type`) because commands and events are
588/// different directions on the wire; the ops console's protocol parser keys command frames on
589/// `command` and event frames on `type`.
590#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
591#[serde(tag = "command")]
592pub enum ClusterCommand {
593    /// Read the current cluster baseline (peers + shards + workers). Read-only; needs only
594    /// namespace-or-deploy scope. **Phase 1.**
595    RequestClusterSnapshot {},
596    /// Cancel a running workflow. Aspirational — handler returns `unimplemented`.
597    CancelWorkflow {
598        /// Owning namespace.
599        namespace: String,
600        /// Target workflow.
601        workflow_id: String,
602    },
603    /// Reopen a failed/closed workflow. Aspirational — handler returns `unimplemented`.
604    ReopenWorkflow {
605        /// Owning namespace.
606        namespace: String,
607        /// Target workflow.
608        workflow_id: String,
609    },
610    /// Redrive a single outbox row. Aspirational — handler returns `unimplemented`.
611    RedriveOutboxRow {
612        /// Owning namespace.
613        namespace: String,
614        /// Target workflow.
615        workflow_id: String,
616        /// Outbox row ordinal.
617        ordinal: u64,
618    },
619    /// Drain a node (stop-new-work + finish-in-flight + safe shutdown). Aspirational.
620    DrainNode {
621        /// Target node name.
622        node: String,
623    },
624    /// Planned epoch-fenced shard handoff to a target node. Aspirational.
625    PlannedHandoff {
626        /// Shard to move.
627        shard: usize,
628        /// Destination node.
629        target_node: String,
630    },
631    /// Test-only chaos kill of a node. Aspirational (gated).
632    ChaosKillNode {
633        /// Target node name.
634        node: String,
635    },
636}
637
638/// A typed terminal error on the cluster channel.
639///
640/// Mirrors the workflow path's lagged contract: when a subscriber falls behind the bounded cluster
641/// broadcast buffer the server sends exactly one of these and closes, carrying the skipped count so
642/// the client can decide snapshot-vs-resume (it always re-requests a [`ClusterSnapshot`], since
643/// there is no durable cluster history). Surfaced to the UI as a typed error, never silently
644/// dropped (ADR-016).
645#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
646#[serde(tag = "kind")]
647pub enum ClusterStreamError {
648    /// The subscriber lagged past the bounded broadcast buffer; `skipped` deltas were dropped.
649    ClusterLagged {
650        /// Number of cluster events dropped because the subscriber fell behind.
651        ///
652        /// Exported to TypeScript as `number`; see the module docs for the accepted `2^53` ceiling.
653        skipped: u64,
654    },
655}