aion_core/cluster_event.rs
1//! Cluster topology and ownership events for the ops console real-time channel (WS3).
2//!
3//! This module defines the *typed contract* for the second class of real-time push the ops console
4//! consumes: cluster/topology/ownership deltas that today exist only as `tracing` logs and
5//! Prometheus gauges. The wire shapes live in `aion-core` (not `aion-server`) for one reason: only
6//! this leaf crate depends on `ts-rs`, so this is the single place a Rust type can cross the
7//! Rust -> TypeScript boundary into the ops console's generated bindings.
8//!
9//! # Scope (WS3 FOUNDATION)
10//!
11//! This file is *types only*. It deliberately does **not** define the broadcast publisher, the
12//! emit call-sites in the supervisor / registry, the subscription endpoint, or the namespace gate.
13//! Those are wired in later WS3 increments. Defining the contract first lets the ops console and the
14//! server agree on shapes before any behaviour ships.
15//!
16//! # Honesty corrections applied to the original design
17//!
18//! The design table named several variants whose payloads cannot be sourced from the subsystems
19//! that would emit them today. Rather than ship a type whose fields are guaranteed to be faked,
20//! those variants are **descoped from Phase 1** and documented here so the omission is explicit and
21//! not silently rediscovered at wiring time:
22//!
23//! - **`NodeMetricsSampled` — DEFERRED.** `aion-server`'s metrics (`observability/metrics.rs`) have
24//! no `node` dimension (`connected_workers` is an `IntGaugeVec` keyed by *namespace*) and there
25//! is no `workflows_running` gauge at all. A `{ node, connected_workers, workflows_running }`
26//! payload cannot be produced by "piggybacking the gauge setters". Sourcing it honestly requires
27//! a real metrics change (node-labelled gauges + a running gauge). Until then the ops console
28//! derives `connected_workers` client-side from [`ClusterEvent::WorkerConnected`] /
29//! [`ClusterEvent::WorkerDisconnected`] deltas against the [`ClusterSnapshot`] baseline. Adding
30//! a timer that scrapes Prometheus to fill this variant is the exact polling-as-push regression
31//! WS3 exists to remove, so the variant is omitted rather than tempting that shortcut.
32//! - **`ShardOwnerChanged` / `FencedCasRejected` — DEFERRED (fast-follow).** The haematite seam
33//! (`HaematiteStore::publish_ln(shard)`) carries only `shard`; the fenced-CAS reject is a
34//! `haematite::DatabaseError::Fenced { .. }` destructured with `..` and mapped to
35//! `StoreError::NotOwner { shard }`, carrying only `shard` — never the owner name or the
36//! attempted/current epochs. The proposed payloads require enriching haematite's `Fenced` error
37//! to surface owner identity and epoch, which is a cross-crate (likely cross-repo) change. The
38//! cluster map is still *honest* without these two: shard adoption is fully observable from the
39//! supervisor's `tick()` ([`ClusterEvent::ShardAdopted`] and friends). Only the store-side
40//! CAS-reject detail is delayed.
41//!
42//! # u64 precision across the TS boundary
43//!
44//! The ts-rs config exports every `u64` as TS `number` (`with_large_int("number")` in
45//! `generated_types.rs`), which truncates above `2^53`. [`ClusterEventMeta::cluster_seq`] and the
46//! epoch fields below are `u64`. This is the *same* accepted ceiling that already applies to
47//! [`crate::EventEnvelope::seq`]; cluster sequencing follows the established project convention
48//! rather than introducing a divergent string encoding. The ceiling is documented on each field so
49//! the gap-detection math on the client is aware of the bound. A long-lived deployment must keep
50//! `cluster_seq` below `2^53`; in practice this is never reached for a topology-event counter.
51
52use chrono::{DateTime, Utc};
53use serde::{Deserialize, Serialize};
54
55use crate::{ActivityId, WorkflowId};
56
57/// Metadata stamped on every [`ClusterEvent`], mirroring [`crate::EventEnvelope`] for the cluster
58/// channel.
59#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
60pub struct ClusterEventMeta {
61 /// Monotonic sequence number assigned by the cluster publisher (a single deployment-global
62 /// `AtomicU64`). The client uses this for gap detection and `after_seq` reconnect dedup, the
63 /// cluster analog of [`crate::EventEnvelope::seq`].
64 ///
65 /// Exported to TypeScript as `number`; see the module docs for the accepted `2^53` ceiling.
66 pub cluster_seq: u64,
67 /// UTC wall-clock instant at which the originating subsystem observed the state change.
68 pub observed_at: DateTime<Utc>,
69}
70
71/// Transport a connected worker is delivered to over.
72///
73/// Mirrors the live `aion_server::worker::registry::WorkerDelivery` discriminants without carrying
74/// the (non-serializable) delivery channels, so it can cross the wire and the ts-rs boundary.
75#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
76#[serde(tag = "transport")]
77pub enum WorkerTransport {
78 /// gRPC bidirectional-stream delivery (the default transport).
79 Grpc,
80 /// Liminal server-push delivery (feature-gated on the server).
81 Liminal,
82}
83
84impl WorkerTransport {
85 /// The word an operator sees for this transport in a log line.
86 ///
87 /// Lives beside the discriminants rather than in the server, so the name a
88 /// liveness withdrawal prints and the name a cluster event carries cannot
89 /// disagree about which link an operator should go and look at.
90 #[must_use]
91 pub const fn name(self) -> &'static str {
92 match self {
93 Self::Grpc => "grpc",
94 Self::Liminal => "liminal",
95 }
96 }
97}
98
99/// Why a worker left the connected set.
100///
101/// NOTE (wiring honesty): the single registry deregistration site
102/// (`ConnectedWorkerRegistry::deregister` / `remove_worker`) does not today distinguish a transport
103/// disconnect from an idle timeout from an explicit deregister. The emit increment MUST derive the
104/// reason at the call site from real signal (e.g. a closed delivery channel vs an explicit
105/// deregister RPC vs a liveness-timeout sweep) or collapse to the variant it can actually prove.
106/// This enum defines the *contract*; it must not be populated with a fabricated distinction.
107#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
108#[serde(tag = "reason")]
109pub enum WorkerDeathReason {
110 /// The worker's delivery transport dropped (stream/connection closed).
111 Disconnect,
112 /// The worker was removed by a liveness/idle-timeout sweep.
113 Timeout,
114 /// The worker explicitly deregistered.
115 Deregistered,
116}
117
118/// Operator-requested lifecycle state for a durable worker deployment.
119#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
120#[serde(rename_all = "kebab-case")]
121pub enum DesiredState {
122 /// The deployment should run.
123 Running,
124 /// The deployment should remain stopped.
125 Stopped,
126}
127
128impl DesiredState {
129 /// Stable human-readable state token.
130 #[must_use]
131 pub const fn token(self) -> &'static str {
132 match self {
133 Self::Running => "running",
134 Self::Stopped => "stopped",
135 }
136 }
137}
138
139/// Create-or-replace result exposed on the cluster and HTTP wires.
140#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
141#[serde(rename_all = "kebab-case")]
142pub enum PutOutcome {
143 /// No record previously existed for the key.
144 Created,
145 /// A record already existed for the key and was replaced.
146 Replaced,
147}
148
149/// Deployment-record lookup state captured when a worker registers.
150#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Copy, Debug, PartialEq, Eq)]
151#[serde(rename_all = "kebab-case")]
152pub enum DeploymentAssociation {
153 /// The named deployment record existed.
154 Known,
155 /// The deployment store was checked and the named record was absent.
156 Absent,
157 /// No deployment store was attached, so existence was not checked.
158 Unchecked,
159}
160
161/// A cluster/topology/ownership delta pushed over the ops console real-time channel.
162///
163/// Tagged union on `type` to match the existing [`crate::Event`] wire shape. Every variant carries
164/// [`ClusterEventMeta`] as `meta`. Cluster events are deployment-scoped, not namespace-stamped at
165/// the envelope level; the `Worker*` variants carry their own `namespaces` so the server-side gate
166/// can intersect them against the caller's grants (see the deferred `cluster_filter`).
167#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
168#[serde(tag = "type")]
169pub enum ClusterEvent {
170 /// A peer was added to this node's watch set (cluster-membership grew mid-session).
171 ///
172 /// Distinct from [`Self::PeerConnected`]: this is a *topology* change (a new peer to watch),
173 /// not a *liveness* transition. Without it, a peer that joins after the priming
174 /// [`ClusterSnapshot`] would silently never appear on the map until the next snapshot.
175 PeerAdded {
176 /// Cluster-event metadata.
177 meta: ClusterEventMeta,
178 /// The newly-watched peer's node name.
179 peer_name: String,
180 /// The peer's forwarding address, when known.
181 forward_addr: Option<String>,
182 },
183 /// A watched peer transitioned from down/unknown to connected (a liveness recovery).
184 ///
185 /// EMIT NOTE: the supervisor `tick()` resets `consecutive_down`/`adopted` on the same tick it
186 /// observes `connected`, so the recovery condition (`was_down = consecutive_down > 0 ||
187 /// adopted`) MUST be captured *before* that reset and the emit driven by the captured value.
188 /// Emitting after the reset produces no recovery events (every tick looks freshly connected).
189 PeerConnected {
190 /// Cluster-event metadata.
191 meta: ClusterEventMeta,
192 /// The recovered peer's node name.
193 peer_name: String,
194 /// The peer's forwarding address, when known.
195 forward_addr: Option<String>,
196 },
197 /// A watched peer was observed down on a supervisor tick.
198 ///
199 /// Emitted on every tick the peer is observed down; `confirmed` flips to `true` once
200 /// `consecutive_down >= confirmations` (the debounce threshold that authorizes adoption).
201 PeerDisconnected {
202 /// Cluster-event metadata.
203 meta: ClusterEventMeta,
204 /// The down peer's node name.
205 peer_name: String,
206 /// Consecutive ticks this peer has been observed down.
207 consecutive_down: u32,
208 /// Whether the debounce threshold has been crossed (adoption-eligible).
209 confirmed: bool,
210 },
211 /// This node adopted shards previously owned by a failed peer.
212 ShardAdopted {
213 /// Cluster-event metadata.
214 meta: ClusterEventMeta,
215 /// Shard indices adopted in this transition.
216 shards: Vec<usize>,
217 /// The peer the shards were adopted from.
218 from_peer: String,
219 /// This node's name (the new owner).
220 adopted_by: String,
221 },
222 /// An adoption attempt failed and will be retried on a subsequent tick.
223 ///
224 /// HONESTY: the original design hardcoded `will_retry: true`. That is a silent lie — a peer
225 /// later observed connected resets `adopted` and stops retrying, and the
226 /// "handled elsewhere" path terminally stops. The retry decision is not expressible as a
227 /// constant, so the field is omitted; the ops console infers retry by observing whether a
228 /// subsequent [`Self::ShardAdopted`] / [`Self::ShardAdoptionSkipped`] for the same
229 /// `(from_peer, shards)` arrives, rather than trusting a promise that may never be kept
230 /// (ADR-016 no-silent-failure).
231 ShardAdoptionFailed {
232 /// Cluster-event metadata.
233 meta: ClusterEventMeta,
234 /// Shard indices the failed attempt targeted.
235 shards: Vec<usize>,
236 /// The peer the shards would have been adopted from.
237 from_peer: String,
238 /// Human-readable adoption error.
239 error: String,
240 },
241 /// Adoption was skipped because the shards are already held by a live third-party owner.
242 ShardAdoptionSkipped {
243 /// Cluster-event metadata.
244 meta: ClusterEventMeta,
245 /// Shard indices that were skipped.
246 shards: Vec<usize>,
247 /// The peer the shards would have been adopted from.
248 from_peer: String,
249 /// The live node currently holding the shards.
250 held_by: String,
251 },
252 /// A worker joined the connected set.
253 WorkerConnected {
254 /// Cluster-event metadata.
255 meta: ClusterEventMeta,
256 /// Stable worker identifier.
257 worker_id: String,
258 /// Namespaces this worker serves.
259 namespaces: Vec<String>,
260 /// Task-queue pool this worker serves within its namespaces.
261 task_queue: String,
262 /// Delivery transport for this worker.
263 transport: WorkerTransport,
264 /// Locality/node label, when the worker reported one.
265 node: Option<String>,
266 /// Deployment named by the worker's instance identity, when supplied.
267 deployment: Option<String>,
268 /// Lookup state for the named deployment. `None` means the worker
269 /// supplied no instance identity.
270 deployment_association: Option<DeploymentAssociation>,
271 },
272 /// A worker left the connected set.
273 WorkerDisconnected {
274 /// Cluster-event metadata.
275 meta: ClusterEventMeta,
276 /// Stable worker identifier.
277 worker_id: String,
278 /// Namespaces this worker served.
279 namespaces: Vec<String>,
280 /// Why the worker left (see [`WorkerDeathReason`] wiring note).
281 reason: WorkerDeathReason,
282 },
283 /// A dispatch parked on an unserved queue with **no availability
284 /// deadline** — the wait that can hold a run forever.
285 ///
286 /// Every other selection outcome has a terminal echo the operator can see:
287 /// a served dispatch completes, and a deadline-bounded park expires into a
288 /// typed `WorkerUnavailable` refusal that reaches the workflow's own
289 /// history. The unbounded park had only a WARN line and the queryable
290 /// unserved state — nothing pushed. This variant is that push (#266 T4):
291 /// the #266 boot-ordering defect stranded recovered runs in exactly this
292 /// state, and the only sign was a log nobody was reading. The status-truth
293 /// front (#264) consumes it.
294 ///
295 /// Emitted under the same transition rule as the WARN it accompanies: once
296 /// when a park episode is first classified, and again only when the
297 /// classified reason changes. Re-observations of the same reason are
298 /// silent, so a queue nobody serves does not flood the channel.
299 ///
300 /// `reason` and `policy` carry the queue-service taxonomy's canonical
301 /// spellings (e.g. `NO_LIVE_POLLERS`, `durable_pending`) as strings rather
302 /// than server-crate enums, the same leaf-crate discipline
303 /// [`Self::NamespaceCreated`] applies to `origin`.
304 DispatchParked {
305 /// Cluster-event metadata.
306 meta: ClusterEventMeta,
307 /// Namespace the dispatch belongs to.
308 namespace: String,
309 /// Task queue the dispatch is addressed to.
310 task_queue: String,
311 /// Activity type the dispatch needs served.
312 activity_type: String,
313 /// Node label the dispatch is pinned to, if any.
314 node: Option<String>,
315 /// Canonical queue-service reason spelling (`NO_QUEUE_DECLARATION` /
316 /// `NO_LIVE_POLLERS` / `POLLERS_INCOMPATIBLE` / `POLLERS_UNREACHABLE` /
317 /// `SATURATED`).
318 reason: String,
319 /// Canonical queue-service policy spelling (`strict` /
320 /// `durable_pending`).
321 policy: String,
322 /// Workflow whose activity is parked.
323 workflow_id: WorkflowId,
324 /// Parked activity's history ordinal.
325 activity_id: ActivityId,
326 /// How long the dispatch had been waiting when this was emitted.
327 ///
328 /// Exported to TypeScript as `number`; see the module docs for the
329 /// accepted `2^53` ceiling.
330 waited_ms: u64,
331 /// Workers connected for `(namespace, task_queue)`, whatever they
332 /// serve, at the census behind this verdict.
333 workers_in_pool: usize,
334 /// Of those, workers advertising this activity type.
335 workers_serving_activity: usize,
336 /// Of those, workers that also satisfy the dispatch's node pin.
337 compatible_workers: usize,
338 /// How long ago a compatible worker was last in service, or `None`
339 /// when none has ever been seen for this address in this server's
340 /// life.
341 ///
342 /// Exported to TypeScript as `number`; see the module docs for the
343 /// accepted `2^53` ceiling.
344 last_compatible_poller_age_ms: Option<u64>,
345 },
346 /// The cluster supervisor started on this node (lifecycle).
347 ///
348 /// Lets the calm-state view (ADR-019) distinguish "supervisor running, all peers healthy" from
349 /// "supervisor not running".
350 SupervisorStarted {
351 /// Cluster-event metadata.
352 meta: ClusterEventMeta,
353 /// This node's name.
354 node: String,
355 },
356 /// The cluster supervisor stopped on this node (clean drain / shutdown).
357 SupervisorStopped {
358 /// Cluster-event metadata.
359 meta: ClusterEventMeta,
360 /// This node's name.
361 node: String,
362 },
363 /// A brand-new namespace was minted in the durable registry (Control-Plane
364 /// Phase 1, S8).
365 ///
366 /// Emitted exactly once per genuinely-new namespace at the registry's single
367 /// `MintOutcome::Created` choke-point — so the worker-register seam (S5), the
368 /// workflow-start safety net (S6), and the explicit `POST /namespaces` path
369 /// (S7) all surface the same delta. An idempotent re-reference of an existing
370 /// namespace (`MintOutcome::AlreadyExisted`) never emits this, so the ops
371 /// console appends each namespace exactly once with no refresh.
372 ///
373 /// `origin` is the stable `snake_case` label (`worker_mint` / `start_mint` /
374 /// `explicit` / `inferred_from_state`) rather than the `aion-store`
375 /// `NamespaceOrigin` enum, because this leaf crate must not depend on the
376 /// store crate; the label is the same string the audit `tracing` event logs.
377 /// `created_at` is the durable record's first-mint instant, carried so the
378 /// console's created column matches the registry without a follow-up fetch.
379 NamespaceCreated {
380 /// Cluster-event metadata.
381 meta: ClusterEventMeta,
382 /// The minted namespace's name (registry primary key).
383 name: String,
384 /// The durable record's creation instant (its `created_at`).
385 created_at: DateTime<Utc>,
386 /// How the namespace came to exist, as the stable `snake_case` label.
387 origin: String,
388 },
389 /// A namespace's durable placement directive was changed (Control-Plane
390 /// Phase 2, P2-P2 — `PUT /namespaces/{name}/placement`).
391 ///
392 /// Emitted on the SAME deploy-scoped channel as [`Self::NamespaceCreated`]
393 /// after the placement is durably set, so the ops console's namespace panel
394 /// reflects an operator's placement change live with no refresh. `placement`
395 /// is the stable wire projection ([`NamespacePlacementWire`]) of the durable
396 /// `NamespacePlacement`, carried as a `kind` label + label set rather than the
397 /// `aion-store` enum, because this leaf crate must not depend on the store
398 /// crate (the same discipline [`Self::NamespaceCreated`] applies to `origin`).
399 NamespacePlacementChanged {
400 /// Cluster-event metadata.
401 meta: ClusterEventMeta,
402 /// The namespace whose placement changed (registry primary key).
403 name: String,
404 /// The new placement directive, as the stable wire projection.
405 placement: NamespacePlacementWire,
406 },
407 /// A periodic snapshot of a namespace's concurrency-quota state (Control-Plane
408 /// Phase 2, P2-Q3), pushed on the SAME deploy-scoped channel as
409 /// [`Self::NamespaceCreated`] so the ops console renders a live "in-flight /
410 /// ceiling" badge that ticks as work flows.
411 ///
412 /// This is the ONE edge-triggered exception's honest counterweight: unlike the
413 /// other variants (each emitted at a real subsystem mutation), quota state
414 /// changes on every claim/settle, so per-row emission would be a firehose.
415 /// Instead a throttled snapshot task samples each active namespace's REAL
416 /// durable state once per cadence and emits this — a periodic snapshot per
417 /// active namespace, never a client poll. The console folds the latest snapshot
418 /// per namespace, superseding the prior value.
419 ///
420 /// `in_flight` is the durable **Claimed** outbox-row count for the namespace
421 /// (`count_claimed_outbox_rows`) — the SAME notion the keyed backpressure caps,
422 /// never the dead `inflight_activities` gauge (P2-Q0). `ceiling` is the tenant's
423 /// **cluster-wide** contract: its explicit `max_in_flight_activities` override
424 /// or the platform default — the number the operator set, not the per-node
425 /// proportional slice (exposing per-node math is the leaky-abstraction footgun
426 /// §3.6 rejects). On a single node the durable `in_flight` this node sees is the
427 /// whole cluster's; multi-node exact aggregation is the reserved §3.6 follow-up.
428 NamespaceQuotaState {
429 /// Cluster-event metadata.
430 meta: ClusterEventMeta,
431 /// The namespace this quota snapshot describes (registry primary key).
432 namespace: String,
433 /// Durable count of currently-**Claimed** outbox rows for the namespace —
434 /// the in-flight activities the ceiling caps.
435 ///
436 /// Exported to TypeScript as `number`; see the module docs for the accepted
437 /// `2^53` ceiling (an in-flight count never approaches it).
438 in_flight: u64,
439 /// The tenant's cluster-wide concurrency ceiling: its explicit
440 /// `max_in_flight_activities` override, or the platform default when unset.
441 ceiling: u32,
442 },
443 /// A worker deployment was durably created or replaced by an operator.
444 WorkerDeploymentPut {
445 /// Cluster-event metadata.
446 meta: ClusterEventMeta,
447 /// Durable deployment name.
448 name: String,
449 /// Typed create-or-replace outcome.
450 outcome: PutOutcome,
451 /// Typed desired state.
452 desired_state: DesiredState,
453 /// Version captured from the deployed server binary.
454 binary_version: String,
455 /// SHA-256 captured from the deployed server executable bytes.
456 binary_content_hash: String,
457 },
458 /// An operator durably changed a deployment's desired state.
459 WorkerDeploymentDesiredStateChanged {
460 /// Cluster-event metadata.
461 meta: ClusterEventMeta,
462 /// Durable deployment name.
463 name: String,
464 /// Typed desired state.
465 desired_state: DesiredState,
466 },
467 /// An operator durably deleted a deployment record.
468 WorkerDeploymentDeleted {
469 /// Cluster-event metadata.
470 meta: ClusterEventMeta,
471 /// Deleted deployment name.
472 name: String,
473 },
474}
475
476/// Stable wire projection of a namespace's durable placement directive
477/// (`NamespacePlacement` in `aion-store`), for the ops-console real-time channel
478/// (Control-Plane Phase 2).
479///
480/// Lives in this leaf crate (not `aion-server` / `aion-store`) for the same reason
481/// the rest of [`ClusterEvent`] does: only this crate crosses the Rust ->
482/// TypeScript boundary via `ts-rs`. `kind` is the stable `snake_case` variant tag
483/// (`unplaced` / `prefer` / `pinned`) and `nodes` is the (possibly empty)
484/// node-label set; `Unplaced` carries an empty `nodes`. Modelling it as a flat
485/// `{kind, nodes}` shape rather than re-encoding the store's tagged form keeps the
486/// generated TS binding a single simple type the console can switch on.
487#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
488pub struct NamespacePlacementWire {
489 /// Stable `snake_case` placement-kind tag: `unplaced` / `prefer` / `pinned`.
490 pub kind: String,
491 /// The placement's node-label set, deterministically ordered. Empty for
492 /// `unplaced`; the preferred/required labels for `prefer`/`pinned`.
493 pub nodes: Vec<String>,
494}
495
496/// A peer entry in the priming [`ClusterSnapshot`].
497#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
498pub struct ClusterPeer {
499 /// The peer's node name.
500 pub peer_name: String,
501 /// The peer's forwarding address, when known.
502 pub forward_addr: Option<String>,
503 /// Whether the peer is currently observed connected.
504 pub connected: bool,
505 /// Consecutive ticks observed down (0 when connected).
506 pub consecutive_down: u32,
507}
508
509/// A shard ownership entry in the priming [`ClusterSnapshot`].
510#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
511pub struct ClusterShard {
512 /// Shard index.
513 pub shard: usize,
514 /// The node that currently owns the shard.
515 pub owner: String,
516 /// The epoch fence value at which the owner holds the shard.
517 ///
518 /// Exported to TypeScript as `number`; see the module docs for the accepted `2^53` ceiling.
519 pub epoch: u64,
520}
521
522/// A connected-worker entry in the priming [`ClusterSnapshot`].
523#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
524pub struct ClusterWorker {
525 /// Stable worker identifier.
526 pub worker_id: String,
527 /// Namespaces this worker serves (already intersected with the caller's grants by the gate).
528 pub namespaces: Vec<String>,
529 /// Task-queue pool this worker serves.
530 pub task_queue: String,
531 /// Delivery transport for this worker.
532 pub transport: WorkerTransport,
533 /// Locality/node label, when reported.
534 pub node: Option<String>,
535 /// Deployment named by the worker's instance identity, when supplied.
536 pub deployment: Option<String>,
537 /// Lookup state for that deployment. `None` for workers without instance
538 /// identity.
539 pub deployment_association: Option<DeploymentAssociation>,
540}
541
542/// A durable worker-deployment entry in the priming cluster snapshot.
543#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
544pub struct ClusterDeployment {
545 /// Durable deployment name.
546 pub name: String,
547 /// Typed desired state.
548 pub desired_state: DesiredState,
549 /// Version captured at deploy time.
550 pub binary_version: String,
551 /// SHA-256 captured at deploy time.
552 pub binary_content_hash: String,
553}
554
555/// A calm-state baseline of the whole cluster, sent as the priming reply before the live delta
556/// stream so the ops console can render an at-a-glance "all clear" before any [`ClusterEvent`]
557/// arrives (ADR-019). On `cluster_lagged` the client re-requests this rather than replaying a
558/// (non-durable) delta history.
559#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
560pub struct ClusterSnapshot {
561 /// The reading node's own name (self-identity; baselines the `Peer*` deltas which describe
562 /// *other* nodes).
563 pub node: String,
564 /// The `cluster_seq` this snapshot is consistent as-of; the client applies only deltas with a
565 /// strictly greater `cluster_seq`.
566 ///
567 /// Exported to TypeScript as `number`; see the module docs for the accepted `2^53` ceiling.
568 pub as_of_seq: u64,
569 /// Watched peers and their current liveness.
570 pub peers: Vec<ClusterPeer>,
571 /// Shards owned by (or visible to) the reading node.
572 pub shards: Vec<ClusterShard>,
573 /// Connected workers, already gated to the caller's namespaces.
574 pub workers: Vec<ClusterWorker>,
575 /// Durable worker deployments visible under the deploy grant.
576 pub deployments: Vec<ClusterDeployment>,
577}
578
579/// A command the ops console can issue against the cluster channel (ADR-020 command seam).
580///
581/// Defined now for contract coherence. **Phase 1 ships only [`Self::RequestClusterSnapshot`]** (a
582/// read). The mutating variants compile so the contract exists, but their handlers reject with an
583/// `unimplemented` wire error — and, per ADR-020, MUST still run the full auth gate
584/// (`caller.deploy_granted()`) *before* rejecting, so the seam's authorization contract is
585/// exercised now and an `unimplemented` stub is never an auth-bypass-shaped hole.
586///
587/// Tagged on `command` (distinct from [`ClusterEvent`]'s `type`) because commands and events are
588/// different directions on the wire; the ops console's protocol parser keys command frames on
589/// `command` and event frames on `type`.
590#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
591#[serde(tag = "command")]
592pub enum ClusterCommand {
593 /// Read the current cluster baseline (peers + shards + workers). Read-only; needs only
594 /// namespace-or-deploy scope. **Phase 1.**
595 RequestClusterSnapshot {},
596 /// Cancel a running workflow. Aspirational — handler returns `unimplemented`.
597 CancelWorkflow {
598 /// Owning namespace.
599 namespace: String,
600 /// Target workflow.
601 workflow_id: String,
602 },
603 /// Reopen a failed/closed workflow. Aspirational — handler returns `unimplemented`.
604 ReopenWorkflow {
605 /// Owning namespace.
606 namespace: String,
607 /// Target workflow.
608 workflow_id: String,
609 },
610 /// Redrive a single outbox row. Aspirational — handler returns `unimplemented`.
611 RedriveOutboxRow {
612 /// Owning namespace.
613 namespace: String,
614 /// Target workflow.
615 workflow_id: String,
616 /// Outbox row ordinal.
617 ordinal: u64,
618 },
619 /// Drain a node (stop-new-work + finish-in-flight + safe shutdown). Aspirational.
620 DrainNode {
621 /// Target node name.
622 node: String,
623 },
624 /// Planned epoch-fenced shard handoff to a target node. Aspirational.
625 PlannedHandoff {
626 /// Shard to move.
627 shard: usize,
628 /// Destination node.
629 target_node: String,
630 },
631 /// Test-only chaos kill of a node. Aspirational (gated).
632 ChaosKillNode {
633 /// Target node name.
634 node: String,
635 },
636}
637
638/// A typed terminal error on the cluster channel.
639///
640/// Mirrors the workflow path's lagged contract: when a subscriber falls behind the bounded cluster
641/// broadcast buffer the server sends exactly one of these and closes, carrying the skipped count so
642/// the client can decide snapshot-vs-resume (it always re-requests a [`ClusterSnapshot`], since
643/// there is no durable cluster history). Surfaced to the UI as a typed error, never silently
644/// dropped (ADR-016).
645#[derive(Serialize, Deserialize, ts_rs::TS, Clone, Debug, PartialEq, Eq)]
646#[serde(tag = "kind")]
647pub enum ClusterStreamError {
648 /// The subscriber lagged past the bounded broadcast buffer; `skipped` deltas were dropped.
649 ClusterLagged {
650 /// Number of cluster events dropped because the subscriber fell behind.
651 ///
652 /// Exported to TypeScript as `number`; see the module docs for the accepted `2^53` ceiling.
653 skipped: u64,
654 },
655}