Module cancel 

Single-lock run cancellation.

run cancel does three things under one held RunLock: refuse a run that is already in a non-cancelled terminal state, synthesize a terminal node.report for every still-live node, and append run.status: cancelled once. Holding one lock for the whole operation serializes it against other cooperating writers (those that honor the lock) so the node reads and the node-report appends can’t interleave — which is what made the pre-refactor CLI loop both racy and prone to over-reporting cancelled_nodes (it pushed a node id even when the per-node append landed after another process had already settled the node, so the reducer dropped it). Under one lock the node we read is the node we cancel, so the reported count is honest.

This is not crash-atomic: each append_and_apply_unlocked is its own durable append, so a crash or I/O error partway through can leave some nodes cancelled and run.status not yet appended. Recovery is convergent — a re-cancel of an already-Cancelled run scans the still-live stragglers and finishes the job — not transactional rollback.

Two consistency properties beyond the single lock:

• Enumeration and per-node liveness are from the event log, not the projection directory. The node set and each node’s current status are both replayed from events.jsonl (the source of truth) in one streaming pass rather than scanned from nodes/*.json. A node.created can be appended+fsynced while its projection write is crash-interrupted (events.rs documents the log leading the projections); a nodes/ scan would silently drop that node, mark the run cancelled, and let a future rebuild_projections resurrect it as live under a Cancelled run. Walking the log closes that window. Crucially, replaying node.status / node.report to derive each node’s status (rather than trusting read_node_opt) closes a second window: a non-cancel terminal event (e.g. a node.report success: true) fsynced but not yet folded leaves a stale-live projection, and a projection-derived liveness check would over-write that node with a fresh cancel that diverges on rebuild. The log-derived status settles it as already-terminal instead — the log wins. (The manifest’s node_count is also a projection written in the same interrupted fold, so it is no more authoritative than nodes/ — and it carries no node ids — which is why we replay the log rather than trust the counter.)

• Each synthesized event carries a deterministic idempotency key (run-cancel:<run_id>:node:<node_id> and run-cancel:<run_id>:run-status). If a crash lands an append+fsync but interrupts the projection fold, the node/run still reads non-terminal, so a re-cancel would append a second logical-cancel event (duplicating it for auditors, metrics, and rebuild). The prior cancel events (scoped by (kind, key) for this run) are captured in the same replay pass, so instead of re-appending, the loop re-folds the already-logged event via apply_event — converging a projection the crash left non-terminal without a duplicate log line (a re-fold is a clean no-op when the projection already agrees). The whole transaction is then both non-duplicating and projection-convergent.

The cancel ledger is built by a streaming pass: for_each_event_probe walks events.jsonl line by line, parsing only the small envelope + status fields each line needs and materializing a full Event payload solely for the handful of lines in this run’s run-cancel:<run_id>: key namespace (the events the re-fold path replays). The whole log is never held in memory, so lock-hold time and peak memory stay bounded even for a run with hundreds of nodes and multi-KB node.report payloads.

What is still not derived from the log here: run-level liveness (the terminal-refusal check and run_was_already_cancelled) is read from the manifest projection, with the prior-cancel re-fold converging a crash-stranded run.status. Deriving the run status from the log too would conflate a crash-stranded run.status: cancelled (manifest stale, must re-fold and report a fresh cancel) with an already-folded one, since the log is identical in both cases — so the manifest read stays authoritative for the run-level decision, exactly as the per-node convergence path consults read_node_opt only to tell those two cases apart.

Structs

CancelOutcome

Outcome of a cancel_run transaction. Lets a thin CLI wrapper report honestly what actually changed: which live nodes it converged, which were already settled (skipped, not double-reported), and whether the run itself was already cancelled (a convergence-only no-op rather than a fresh cancel).

Functions

cancel_run

Cancel a run in a single locked transaction. Acquires the run’s RunLock once for the whole operation, then delegates to cancel_run_unlocked.

cancel_run_unlocked

The locked body of cancel_run. The lock: &LockedRun witness proves the caller already holds the run’s exclusive RunLock; this is the sanctioned lock-held composition path so the manifest read, the per-node read-then-append loop, and the final run.status append all share one critical section (it calls append_and_apply_unlocked, never crate::append_and_apply_event, which would deadlock by re-locking).