Expand description
Restart recovery: reload persisted non-terminal agents into a fresh world when
the daemon starts, so runs interrupted by a stop/crash resume where they left
off - critically, any agent that was mid-inference re-issues that inference
(the reloaded agent is ReadyToInfer), rather than being lost.
For each <runs_dir>/<run_id>/meta.json whose status is non-terminal, this
loads the blueprint (via build_agent_for_reload, reusing the spawn path),
which skips the required-at-spawn region gate since the window is restored
from a snapshot; restores the
persisted context / stage / iteration / token totals via
leviath_runtime::restore::restore_agent, puts the run’s per-stage ledger
back from stages.json via
leviath_runtime::restore::restore_stage_ledger, and preserves the
original run metadata. Anything unreadable or un-reloadable is skipped (logged), never fatal.
One exception to the “re-issue inference” resume: a run that was parked at a
stage-boundary interaction point (e.g. plan_approval) wrote an
interactions.json sidecar while blocked. For those, reload_one calls
leviath_runtime::interaction_points::restore_interaction_point to bring the
agent back in the waiting state with the same prompt re-opened, rather than
re-inferring and dropping it. Model-initiated dynamic tools
(ask_user_*, present_for_review, edit_document) and taint-gate prompts are
not persisted - they block inside the transient tool-worker turn, so on restart
they take the ordinary re-inference path and the model simply re-asks.
§Tool-call delivery contract (issue #96)
A tool batch in flight at the crash is replayed, not re-executed. Dispatch
journals the batch (a ToolBatch record) before its side effects can start,
and every call’s result the moment it finishes (ToolCallDone); when the fold
surfaces such a pending batch, reload_one calls
leviath_runtime::restore::restore_pending_batch to land the assistant turn
with each completed call’s real journaled result - so completed side effects
are exactly-once across a restart. Calls whose completion never reached the
journal (still executing, or the crash landed in the instant between the
external effect and its journal append - a window no journal can close,
since an external side effect can’t be observed atomically) come back as
verify-first [error] interrupted results rather than being silently re-run;
the re-issued inference decides what still needs doing.
Functions§
- reload_
persisted_ agents - Reload every non-terminal persisted run under
runs_dir, returning the(run_id, entity)pairs for the host to map. Runs that fail to reload are skipped. - reload_
run - Page a single unloaded run back into the world from disk, on demand. Reads
its persisted metadata; if the run exists and is non-terminal, reloads it
(blueprint + tool state + context/stage) and returns the new entity.
Noneif there’s no such resumable run. This is the host’s reload-on-demand seam (an op targeting an unloaded run pages it in first).