Skip to main content

Module recovery

Module recovery 

Source
Expand description

Restart recovery: reload persisted non-terminal agents into a fresh world when the daemon starts, so runs interrupted by a stop/crash resume where they left off - critically, any agent that was mid-inference re-issues that inference (the reloaded agent is ReadyToInfer), rather than being lost.

For each <runs_dir>/<run_id>/meta.json whose status is non-terminal, this loads the blueprint (via build_agent_for_reload, reusing the spawn path), which skips the required-at-spawn region gate since the window is restored from a snapshot; restores the persisted context / stage / iteration / token totals via leviath_runtime::restore::restore_agent, puts the run’s per-stage ledger back from stages.json via leviath_runtime::restore::restore_stage_ledger, and preserves the original run metadata. Anything unreadable or un-reloadable is skipped (logged), never fatal.

One exception to the “re-issue inference” resume: a run that was parked at a stage-boundary interaction point (e.g. plan_approval) wrote an interactions.json sidecar while blocked. For those, reload_one calls leviath_runtime::interaction_points::restore_interaction_point to bring the agent back in the waiting state with the same prompt re-opened, rather than re-inferring and dropping it. Model-initiated dynamic tools (ask_user_*, present_for_review, edit_document) and taint-gate prompts are not persisted - they block inside the transient tool-worker turn, so on restart they take the ordinary re-inference path and the model simply re-asks.

§Tool-call delivery contract (issue #96)

A tool batch in flight at the crash is replayed, not re-executed. Dispatch journals the batch (a ToolBatch record) before its side effects can start, and every call’s result the moment it finishes (ToolCallDone); when the fold surfaces such a pending batch, reload_one calls leviath_runtime::restore::restore_pending_batch to land the assistant turn with each completed call’s real journaled result - so completed side effects are exactly-once across a restart. Calls whose completion never reached the journal (still executing, or the crash landed in the instant between the external effect and its journal append - a window no journal can close, since an external side effect can’t be observed atomically) come back as verify-first [error] interrupted results rather than being silently re-run; the re-issued inference decides what still needs doing.

Functions§

reload_persisted_agents
Reload every non-terminal persisted run under runs_dir, returning the (run_id, entity) pairs for the host to map. Runs that fail to reload are skipped.
reload_run
Page a single unloaded run back into the world from disk, on demand. Reads its persisted metadata; if the run exists and is non-terminal, reloads it (blueprint + tool state + context/stage) and returns the new entity. None if there’s no such resumable run. This is the host’s reload-on-demand seam (an op targeting an unloaded run pages it in first).