Expand description
The unattended scheduler and independently leased task workers.
The scheduler chooses tasks and execution lanes; a detached worker owns each queue claim, graph driver and outcome settlement. Replacing the scheduler reconstructs occupied slots from live worker jobs, without touching their attempts, passes or paid agent calls. Finite maintenance jobs follow the same lifetime rule. Explicit parking is durable control, never an upgrade step.
§A crashed worker is legible
A task is written Running, with its run id, before graph execution. The
worker keeps its claim until the outcome is recorded. Recovery requires both
no live worker reservation and an obtainable task claim, and settles the
saved run through the same policy as a live finish. A scheduler’s missing
heartbeat alone is never evidence that a run died.
§Retries are bounded
Every attempt at a task consumes one of Opts::max_attempts, after which
the task is crate::queue::TaskStatus::Held for a human. The one
exception is a run that ended Stalled: the panel collapsed because the
agent CLIs hit their quota, which is a fact about the machine and not about
the task, so it must not spend an attempt. Without that exception a quota
outage would quietly hold the entire backlog, and the operator would come
back to a reset quota and nothing left that the loop is willing to run.
Structs§
- Current
- What the loop is working on, for the status file.
- Opts
- How the loop should behave.
- Reading
- The daemon’s published state, read permissively.
- Status
- The daemon’s liveness, published to
<home>/daemon.json. - Stop
- A cooperative stop, shared with whoever asked the loop to run.
- Verdict
- What a finished run tells the queue about the task it came from.
Constants§
- HEARTBEAT
- How often the status file is refreshed. A reader treats a status file older
than
STALE_SECSas “no daemon”, so the heartbeat has to be brisk enough that a busy daemon is never mistaken for a dead one. - POLL
- Default queue poll interval.
- SCHEMA
- On-disk format for
Status. Bumped when a field’s meaning changes. - STALE_
CLAIM - How old a claim has to be before startup sweeps it. Longer than any run this graph plausibly takes, so a sweep cannot pull a task out from under a daemon that is merely slow.
- STALE_
SECS - How old a heartbeat may be before a reader calls the daemon dead. Six missed beats: long enough to survive a slow filesystem, short enough that a crashed daemon is not still reported as running a task.
- STALLED_
RUNNING - How long a task may sit
TaskStatus::Runningwith no live daemon’s heartbeat naming it beforecrate::conductis shown it as stalled.
Functions§
- apply_
choice_ actions - Apply answered typed choices. Only the daemon calls this in production; deputies settle the question and never execute its actions.
- clear_
status - Delete the status file. Called on the way out so a clean exit reads as “no daemon” rather than as a daemon whose heartbeat merely stopped.
- current_
work - Read-only evidence of runs owned by a live daemon or independent worker.
- finish_
attempt - Settle
taskafter its run executed: the verdict, the diagnosis, the supersession of earlier attempts and the save. Shared by the loop and bymagi run --resume, so an attempt is counted one way wherever it ran. Returns the quota losses this execution added to the run. - foreign_
loop - A loop in another process that is alive right now: a fresh heartbeat in
<home>/daemon.jsonpublished by a pid that is notown_pid.Some(pid)carries the pid it published (Noneinside when it published none - a fresh heartbeat is still evidence of a live loop). - hold_
if_ runnable - A task a hand-started run leaves runnable (failed with attempts to spare,
or requeued) is held: nobody asked for a retry, and a loop starting
later must not spend agent calls on it.
magi task releaseretries. - is_
working_ on - Whether a live daemon is working on this run at this moment.
- is_
working_ on_ short - Whether a live daemon is working on a run whose short id is this one.
- is_
working_ on_ task - Whether a live daemon is working on this task at this moment.
- loop_
would_ resume - Whether an attempt at a task whose newest run is
runwould resume that run rather than start a fresh competition.magi run --resumehands a run to a live loop only when this holds. - managed
- Whether an independent scheduler can be controlled without its web parent.
- read_
status - Read
<home>/daemon.jsonpermissively, orNonewhen there is nothing usable there. - request_
control - Stop dispatch immediately; only an explicit park reaches existing workers.
- restart_
detached - Replace the scheduler, leaving all active task and conversation jobs alone.
- run_
claimed - Low-level claimed-task execution shared by the detached standalone worker
and focused lifecycle fixtures. The caller must retain the queue claim for
this entire call. CLI callers transfer ownership through
run_detached_claimedinstead of executing inside their own runtime. - run_
detached_ claimed - Transfer a standalone caller’s claim only after the detached worker is live, then follow that worker to its final (non-automatically-retrying) outcome.
- run_
scheduler_ worker - Hidden worker dispatch entry. The worker infrastructure owns its job lease.
- run_
service_ worker - Finite maintenance jobs survive scheduler replacement without duplicating agent calls or cancelling an operation between its effect and receipt.
- run_
task_ worker - Execute and settle one task with the queue claim owned by this process.
- serve
- Run the loop until Ctrl-C, or until the queue drains with
Opts::once. - serve_
until serve, but stopping whenstopis set as well as on Ctrl-C.- settle
- Record a finished run against the task it came from.
- spawn_
claimed - Launch the standalone task worker with a claim-to-reservation handoff. Returns only after the worker owns a durable execution reservation.
- spawn_
detached - Start a detached, singleton scheduler using this executable generation.
- status_
path - Where the status file lives.
- supersede_
prior_ runs - Rewrite this task’s now-moot earlier attempts —
Task::superseded_attempts— fromBlocked/StalledtoRunStatus::Superseded, once the task itself has actually finished and its own last attempt is provably why: this loads that run and requiresMerged/Readybefore touching anything earlier inruns, sinceTask::succeedcan also be called directly on a task whose last attempt never landed at all. - sweep_
stale_ claims - Remove claim files whose owner is provably dead, or that have simply
outlived
older_than, and return the task ids swept. - write_
status - Publish the status file for this process.
- write_
status_ to - Publish a status to an explicit path.