Expand description
Superstep executor for the durable graph.
This is the engine that makes the recursive runtime durable: it drives a
CompiledGraph in checkpointed supersteps, and because a node handler may
recurse into another compiled graph (a subgraph) or a sub-agent, every level
of that recursion is observed through the same step/boundary/checkpoint
discipline — child runs roll their state, events, and interrupts up through
the parent’s reducer and checkpointer.
The executor runs in supersteps. Each step: take the active node set, run
each active node against the committed state snapshot, collect updates /
commands / interrupts, apply the reducer at the step boundary, persist a
checkpoint at the boundary (when a checkpointer is configured), then select
the next active set. The loop stops when the active set empties, every
branch reaches END, an interrupt pauses the run, or the recursion limit
is hit (a deterministic TinyAgentsError::RecursionLimit).
By default execution is sequential within a step. When the graph is compiled
with crate::graph::GraphBuilder::with_parallel, a step with more than one
active node runs every branch concurrently via
futures::future::join_all, yet the data flow — snapshot reads, boundary
reducer application, boundary checkpointing — is identical: each branch reads
the same committed snapshot (its own clone), and results are folded into the
reducer in deterministic active-set order at the step boundary, so the merged
state is reproducible regardless of which branch finishes first.
§Concurrency and interrupt semantics
- All active branches in a parallel step start before any is awaited, and all
are driven to completion (
join_all) before the step boundary runs. - Branch results are then folded in active-set index order. The reducer is the fan-in / join: lower-index branches’ updates are applied first.
- The lowest-index branch that errors or interrupts is the step’s terminal outcome. Updates produced by lower-index successful branches are still applied/persisted; an error persists a resumable failure boundary (see below) and aborts, an interrupt persists a checkpoint whose pending nodes are that branch and every later active node.
- Because branches run on cloned snapshots and never share mutable state, concurrency is data-race free; the reducer alone resolves conflicting writes (deterministically, by index).
§Network resilience and resumable failures
Two opt-in mechanisms make a run durable under transient failure and restartable after a hard one:
- Node retry. With
CompiledGraph::with_node_retry, a node whose handler fails with a retryable error (a model or tool error — the transient class) is re-run from its start up to the policy’s attempt cap, emittingGraphEvent::NodeRetryScheduledand sleeping the opt-in backoff between attempts. A single network blip is absorbed without touching the run. - Resumable failure. When a handler fails beyond the retry budget (or the
error is non-retryable), the executor does not discard the step. On a
checkpointed thread it folds the branches that already completed into
committed state and persists a failure-boundary checkpoint whose
next_nodesschedule the failed node (and the not-yet-run tail) for a laterCompiledGraph::resume/CompiledGraph::retry, with the error and failed node stamped into the checkpoint metadata. The run then reportsFailed(carrying that checkpoint id) and returns the error. A caller can restart it as-is, or continue on operator feedback by editing state withCompiledGraph::update_statebefore resuming. Without a checkpointer the run aborts immediately, exactly as before.
Structs§
- Compiled
Graph - An immutable, validated graph ready to execute.
- Graph
Execution - The result of a durable graph run.
- Graph
Input - One external input used to seed a graph run.
- State
Snapshot - A point-in-time view of a thread’s checkpointed state, returned by
CompiledGraph::get_stateandCompiledGraph::get_state_history.
Enums§
- Resume
Target - Selects which checkpoint a time-travel resume starts from.