Skip to main content

magi/
deputy.rs

1//! Deputies: the seat that waits on a conductor's question, so the conductor
2//! never has to - and on land's merge approval, whose free-text reply used to
3//! reach nobody either.
4//!
5//! [`crate::conduct`] files a question for the operator and moves on; it is
6//! non-blocking by construction, because one task's question must not park the
7//! whole polling loop. The price used to be that nothing at all was waiting on
8//! it: the owner's free-text reply landed in a thread no agent read, while the
9//! phone said "waiting for the agent". Question f1dc on task 684d sat open
10//! forever on a "setup done".
11//!
12//! A deputy is the answer. For each open conductor question, `magi serve`
13//! runs one short-lived agent seat that is handed what the conductor knew
14//! ([`brief`]: the task, why it asked, what each option leads to), blocks on
15//! the question with `magi ask --wait`, answers back in the thread with
16//! `magi ask --thread`, and - when the owner's own words clearly pick an
17//! option - records that with `magi ask --settle`. Applying the outcome is not
18//! the deputy's job: an answered conductor question goes through the daemon's
19//! existing `resolve_blockers` / action path, exactly as one tapped on the
20//! phone does.
21//!
22//! # Invariants
23//!
24//! - **One seat per question, keyed by seat name.** `deputy-<question id>`,
25//!   never the conductor's shared seat (`conduct/seat.json`, overwritten by
26//!   every cycle) and never an agent id.
27//! - **Persisted on the question** ([`crate::ask::Deputy`]), always through
28//!   [`Questions::update`]: the owner's say and answer write the same file. A
29//!   restarted daemon resumes the seat only when [`agent::has_session`] says
30//!   it can; otherwise it starts a fresh seat and re-sends the whole context,
31//!   never assuming memory.
32//! - **Bounded.** At most `daemon.max_deputies` at once; at most
33//!   [`MAX_STARTS`] starts per question, and a restart does not reset that. A
34//!   deputy never outlives the question's `answer_timeout`: the waiter retires
35//!   the question at its deadline ([`crate::waiter`]) and the task goes to a
36//!   machine hold (`daemon::resolve_blockers`).
37//! - **Never a second agent on one question.** A fresh lease or the claim file
38//!   means somebody is already on it.
39//! - **A merge approval is served too, but stays land's.** Its deputy has no
40//!   `cwd` (so the waiter never touches it), its deadline is `asked_at +
41//!   answer_timeout` and never moves on a reply ([`deadline`]), and the only
42//!   thing that retires it is `daemon::land_resume_state`. A say alone never
43//!   merges: `--settle` accepts `merge` only for the owner's own word `merge`.
44//! - **A release-watch question is served too, and stays the watcher's.** The
45//!   escalation, local-mode approval and failed-release questions
46//!   ([`crate::release_watch`], node `release-bump`, seat `release-watch`) have
47//!   no asker, so a say reached nobody. Its deputy has no `cwd` either, a
48//!   fixed `asked_at + answer_timeout` deadline ([`deadline`]), and applies
49//!   nothing: `--settle` records a choice and the watcher applies it on its next
50//!   lap. A local approval's `merge` / `hold` go through the merge-approval
51//!   rules ([`merge_gated`]). A deputy never closes or merges the pull request.
52//! - **Every open question the owner can say something to has a listener.**
53//!   [`Kind::Triage`] serves the triage questions; [`Kind::Generic`] is the
54//!   fallback for any other question with choices and no `cwd` (the divergence
55//!   question today, a node nobody has written yet tomorrow). It is decided by
56//!   the missing `cwd`, never by node name. Only a conductor question is ever
57//!   given a `cwd` ([`Deputies::attach`]); the other kinds would otherwise be
58//!   resumed and expired by the waiter beside the deputy. A test enumerates
59//!   every node filed without an asker, so a new kind of question cannot ship
60//!   without one.
61//! - **No new authority.** A deputy does not edit, merge or touch the queue,
62//!   and `--settle` accepts only an offered label backed by a verbatim quote
63//!   of the owner, never on a task the operator holds.
64
65use std::collections::{BTreeMap, HashMap, HashSet};
66use std::path::{Path, PathBuf};
67use std::sync::Arc;
68use std::time::{Duration, Instant};
69
70use anyhow::{Context as _, Result};
71use jiff::Timestamp;
72use tokio::task::JoinSet;
73
74use crate::agent::{self, Invocation, SeatState};
75use crate::ask::{ChoiceAction, Deputy, Question, Questions, Waiter as Note, WaiterKind, Who};
76use crate::config::Config;
77use crate::notices::{self, Notice};
78use crate::prompt;
79
80/// Node name a deputy runs under (`MAGI_NODE`); `magi ask --settle` accepts
81/// it and nothing else.
82pub const NODE: &str = "deputy";
83
84/// How often the runner looks at the store.
85pub const TICK: Duration = Duration::from_secs(5);
86
87/// Most turns ever started for one question. A deputy that keeps ending
88/// without an answer is a problem for the owner, not something to retry until
89/// the quota is gone.
90pub const MAX_STARTS: u32 = 3;
91
92/// Least gap between two starts for one question, including new owner input.
93/// Persisted independently of a runner so finite service laps cannot repeatedly
94/// bill the same conversation or exhaust its lifetime start quota.
95const RESTART_AFTER: Duration = Duration::from_secs(30);
96
97fn next_start_path(store: &Questions, id: &str) -> PathBuf {
98    store.root().join(format!("{id}.deputy")).join("next-start")
99}
100
101fn start_due(store: &Questions, id: &str, now: Timestamp) -> bool {
102    std::fs::read_to_string(next_start_path(store, id))
103        .ok()
104        .and_then(|text| text.parse::<i64>().ok())
105        .is_none_or(|next| now.as_second() >= next)
106}
107
108/// Added to the question's remaining time for the invocation's wall clock, so
109/// the deputy's own `magi ask --wait` reaches the deadline first.
110const SLACK_SECS: u64 = 120;
111
112/// A claim file older than this was left by a process that died.
113const CLAIM_STALE: Duration = Duration::from_secs(60);
114
115/// How long the context-handover turn may take.
116const HANDOVER_TIMEOUT: Duration = Duration::from_secs(10 * 60);
117
118/// Stops the deputy when true: the daemon is parking.
119pub type Halt = Arc<dyn Fn() -> bool + Send + Sync>;
120
121/// What the conductor knew about one question, for its deputy.
122///
123/// `reason` is the conductor's own one-line reasoning (may be empty). Each
124/// choice is listed with what picking it does: an attached action, or - for
125/// the usual conductor question - that the answer is recorded on the task and
126/// the task is unblocked with it in its instructions.
127pub fn brief(
128    task_id: &str,
129    reason: &str,
130    choices: &[String],
131    actions: &BTreeMap<String, ChoiceAction>,
132) -> String {
133    let reason = reason.trim();
134    let mut s = format!(
135        "Task {task_id} (`magi task show {task_id}`) is blocked on this question. \
136         The conductor asked because: {}\n\n\
137         When the owner answers, the daemon applies the exact attached action \
138         once, after rechecking its preconditions. You only record the choice; \
139         you never execute maintenance or edit repository or run records.",
140        if reason.is_empty() {
141            "(it recorded no reasoning)"
142        } else {
143            reason
144        }
145    );
146    if !choices.is_empty() {
147        s.push_str("\n\nWhat each option does:");
148        for c in choices {
149            match actions.get(c) {
150                Some(a) => s.push_str(&format!("\n- `{c}`: also {}", a.describe())),
151                None => s.push_str(&format!(
152                    "\n- `{c}`: no executable operation; keep held and ask for a supported choice"
153                )),
154            }
155        }
156    }
157    s
158}
159
160/// What a deputy serves: the question kinds it is attached to.
161#[derive(Debug, Clone, Copy, PartialEq, Eq)]
162pub enum Kind {
163    /// A conductor's question ([`crate::conduct::NODE`]).
164    Conduct,
165    /// The merge approval ([`crate::land::APPROVAL_NODE`]).
166    Land,
167    /// A question the release watcher filed ([`crate::release_watch`]).
168    Release,
169    /// A triage question about a held task or a stuck dependency root
170    /// ([`crate::triage::NODE`], [`crate::triage::DEPS_NODE`]).
171    Triage,
172    /// Any other question nobody asked from inside a run (no `cwd`) that offers
173    /// choices: the fallback, so a node nobody has met yet still has a listener.
174    Generic,
175}
176
177/// The kind of question a deputy serves for `q`, `None` for every other.
178///
179/// The fallback is decided by the absence of a `cwd`, never by the node name: a
180/// question filed with `magi ask` inside a run records its `cwd` and has an
181/// asker (and the waiter), and the node a seat asks from is the seat's own name
182/// (a reviewer's `review` is also the divergence question's node). A question
183/// with no choices cannot be settled and is a notice, not something to answer.
184pub fn kind_of(q: &Question) -> Option<Kind> {
185    if q.routed_task.is_some() {
186        return None;
187    }
188    match q.node.as_str() {
189        crate::conduct::NODE => Some(Kind::Conduct),
190        crate::land::APPROVAL_NODE => Some(Kind::Land),
191        // The seat matters: `bump` files choice-less notices on the same node,
192        // which nobody can answer and which must not cost a deputy.
193        crate::bump::NOTICE_NODE if q.seat == "release-watch" => Some(Kind::Release),
194        crate::bump::NOTICE_NODE => None,
195        crate::triage::NODE | crate::triage::DEPS_NODE => Some(Kind::Triage),
196        _ if q.cwd.is_none() && !q.choices.is_empty() => Some(Kind::Generic),
197        _ => None,
198    }
199}
200
201/// What a deputy is told about a question of no known kind: only what the
202/// question itself stored.
203pub fn generic_brief(
204    q: &Question,
205    actions: &BTreeMap<String, ChoiceAction>,
206    home: &Path,
207) -> String {
208    // `Question::run` may also hold a task id; only a run id can be shown.
209    let run = crate::run::is_run_id(&q.run).then_some(q.run.as_str());
210    let knows = if run.is_some() {
211        "You know what the question itself says and what you can learn from \
212         its run by the read-only investigation described below."
213    } else {
214        "You know only what the question itself says."
215    };
216    let mut s = format!(
217        "This question was filed by magi (node `{}`, seat `{}`) with no agent \
218         waiting on it. Magi records the owner's answer and the component that \
219         asked applies it; you apply nothing. {knows} A choice whose effect you \
220         cannot read is not yours to guess: do not settle it, ask the owner \
221         what they mean with `--thread` instead.",
222        q.node, q.seat
223    );
224    if let Some(run) = run {
225        let artifacts = home.join("runs").join(run).join("artifacts");
226        s.push_str(&format!(
227            "\n\nThis question concerns run `{run}`. You may investigate it on \
228             your own, read-only: run `magi show {run}` and read files under \
229             `{}`. When the owner asks about details (for example which part \
230             was wrong), look it up yourself and answer; do not reply that you \
231             cannot see the details, and do not tell the owner to run `magi show` \
232             themselves. Investigating never changes anything: edit, commit, push \
233             and apply nothing. If the run cannot be read, say exactly that and do \
234             not guess at the cause or at what an option does.",
235            artifacts.display()
236        ));
237    }
238    if q.node == crate::github_text::ASK_NODE {
239        s.push_str("\n\n");
240        s.push_str(&crate::github_text::deputy_brief());
241    }
242    if !q.choices.is_empty() {
243        s.push_str("\n\nWhat each option does:");
244        for c in &q.choices {
245            match actions.get(c) {
246                Some(a) => s.push_str(&format!("\n- `{c}`: also {}", a.describe())),
247                None => s.push_str(&format!(
248                    "\n- `{c}`: {}",
249                    crate::choice_effect::describe(q, c)
250                        .unwrap_or_else(|| "unsupported legacy choice; do not settle".into())
251                )),
252            }
253        }
254    }
255    s
256}
257
258/// Is picking `label` on `q` something that cannot be taken back, so that
259/// `--settle` must hold the owner's words to the same mechanical standard as a
260/// merge (`land::unhedged`)? Discarding a task deletes it; a divergence answer
261/// drops commits.
262pub fn destructive(q: &Question, label: &str) -> bool {
263    crate::consult::destructive_effect(q, label)
264}
265
266/// Does `--settle` hold `q` to the merge-approval rules (the verbatim quote of
267/// the latest message for `merge`, the whole message for `hold`)? A merge
268/// approval, and any other served question that offers `merge` (the release
269/// watcher's local-mode approval merges just as irreversibly).
270pub fn merge_gated(q: &Question) -> bool {
271    q.node == crate::land::APPROVAL_NODE
272        || (kind_of(q).is_some_and(|k| k != Kind::Conduct)
273            && q.choices.iter().any(|c| c == crate::land::APPROVE))
274}
275
276/// Does `q`'s clock run from `asked_at` and never move on a reply? True for the
277/// questions that something other than the waiter retires: a merge approval
278/// (land) and a release-watch question (the watcher, by silence being a hold).
279pub fn fixed_clock(q: &Question) -> bool {
280    q.node == crate::land::APPROVAL_NODE
281        || (q.node == crate::bump::NOTICE_NODE && q.seat == "release-watch")
282        || q.node == crate::github_text::ASK_NODE
283}
284
285/// Second after which nobody is to be started or kept on `q`.
286///
287/// A conductor question runs from its last activity (`magi ask --thread`
288/// re-arms it, and the waiter retires it). A merge approval never moves: it
289/// runs from `asked_at`, exactly where `daemon::land_resume_state` abandons it,
290/// so a conversation cannot stretch the hold and land stays the only place that
291/// retires one.
292pub fn deadline(q: &Question, default_timeout: u64) -> i64 {
293    let secs = if q.answer_timeout > 0 {
294        q.answer_timeout
295    } else {
296        default_timeout
297    };
298    let from = if fixed_clock(q) {
299        q.asked_at.as_second()
300    } else {
301        q.last_activity()
302    };
303    from.saturating_add(secs as i64)
304}
305
306/// Dispatch and text admission share the fixed-clock and unread-word rules.
307pub fn restart_in_time(q: &Question, default_timeout: u64, now: Timestamp) -> bool {
308    now.as_second() <= deadline(q, default_timeout) || q.unread_from_owner().is_some()
309}
310
311/// Can `magi serve` start a deputy at all under `cfg`? Not when deputies are
312/// switched off (`daemon.max_deputies = 0`) or the config could not be read.
313///
314/// Also false when the agent the deputy would run as cannot be resolved
315/// (`agent` is the deputy's recorded agent, empty when it has none): `turn`
316/// fails on that before it counts a start, so it would never reach
317/// [`MAX_STARTS`].
318pub fn can_start(cfg: Option<&Config>, agent: &str) -> bool {
319    cfg.is_some_and(|c| {
320        c.daemon.max_deputies > 0
321            && ((!agent.is_empty() && c.agent(agent).is_ok()) || c.resolve_roles().is_ok())
322    })
323}
324
325/// The agent a question's deputy records, empty when it has none yet.
326pub fn agent_of(q: &Question) -> &str {
327    q.deputy.as_ref().map_or("", |d| d.agent.as_str())
328}
329
330/// Has this question's deputy run out of starts, or can none ever start, with
331/// the deadline gone?
332///
333/// Then nothing will ever read an unread say, and the waiter must retire the
334/// question anyway instead of deferring to a deputy that no longer starts.
335/// `startable` is [`can_start`]; a fresh lease is the caller's to check.
336pub fn exhausted_past_deadline(
337    q: &Question,
338    startable: bool,
339    default_timeout: u64,
340    now: Timestamp,
341) -> bool {
342    q.status.open()
343        && q.deputy
344            .as_ref()
345            .is_some_and(|d| d.starts >= MAX_STARTS || !startable)
346        && now.as_second() > deadline(q, default_timeout)
347}
348
349/// Question-service dispatcher; each invocation owns an independent worker.
350pub struct Deputies {
351    store: Questions,
352    home: PathBuf,
353    cfg: Option<Config>,
354    /// Working directory for a question that recorded none.
355    fallback_repo: PathBuf,
356    max: usize,
357    halt: Halt,
358    tasks: JoinSet<String>,
359    inflight: HashSet<String>,
360    /// Earliest next start per question.
361    memo: HashMap<String, Instant>,
362}
363
364impl Deputies {
365    /// A runner over `store`, with at most `max` deputies at once.
366    pub fn new(
367        store: Questions,
368        home: PathBuf,
369        cfg: Option<Config>,
370        fallback_repo: PathBuf,
371        max: usize,
372        halt: Halt,
373    ) -> Self {
374        Self {
375            store,
376            home,
377            cfg,
378            fallback_repo,
379            max,
380            halt,
381            tasks: JoinSet::new(),
382            inflight: HashSet::new(),
383            memo: HashMap::new(),
384        }
385    }
386
387    /// Admission check for a finite tick; an idle store needs no child process.
388    pub fn has_work(&self, now: Timestamp) -> bool {
389        self.max > 0
390            && self.store.list().iter().any(|q| {
391                kind_of(q).is_some()
392                    && !crate::task_chat::routes(q, &self.home)
393                    && q.status.open()
394                    && q.deputy
395                        .as_ref()
396                        .is_none_or(|deputy| deputy.starts < MAX_STARTS)
397                    && self
398                        .store
399                        .read_lease(&q.id)
400                        .is_none_or(|lease| !lease.fresh(now))
401                    && can_start(self.cfg.as_ref(), agent_of(q))
402                    && restart_in_time(q, self.default_timeout(), now)
403                    && start_due(&self.store, &q.id, now)
404            })
405    }
406
407    fn default_timeout(&self) -> u64 {
408        self.cfg
409            .as_ref()
410            .map_or(Config::default().graph.answer_timeout, |c| {
411                c.graph.answer_timeout
412            })
413    }
414
415    fn reap(&mut self) {
416        while let Some(done) = self.tasks.try_join_next() {
417            if let Ok(id) = done {
418                self.inflight.remove(&id);
419            }
420        }
421        if self.tasks.is_empty() {
422            self.inflight.clear();
423        }
424    }
425
426    /// Give a question the record a deputy needs: the deputy itself (with a
427    /// brief rebuilt from what the question stored, when it was filed before
428    /// deputies existed - a lost reason is not invented) and the deadline.
429    ///
430    /// A conductor question also gets the working directory the waiter and
431    /// `magi ask --wait` use. Every other kind never does: `cwd` is what makes a
432    /// question the waiter's (it would resume and expire it beside the deputy),
433    /// and a release-watch question, a merge approval, a triage question and a
434    /// fallback one have no asker to resume.
435    fn attach(&self, q: &Question, kind: Kind) -> Option<Question> {
436        let default_timeout = self.default_timeout();
437        let repo = self.fallback_repo.to_string_lossy().into_owned();
438        let state = match kind {
439            Kind::Land => crate::run::RunState::load(&q.run).ok(),
440            Kind::Conduct | Kind::Release | Kind::Triage | Kind::Generic => None,
441        };
442        // The deadline land itself enforces for this run.
443        let timeout = state
444            .as_ref()
445            .map_or(default_timeout, |s| s.config.graph.answer_timeout);
446        self.store
447            .update(&q.id, |r| {
448                if r.deputy.is_none() {
449                    r.deputy = Some(Deputy::new(match kind {
450                        Kind::Conduct => brief(&r.run, &r.detail, &r.choices, &r.actions),
451                        Kind::Land => crate::land::deputy_brief(r, state.as_ref()),
452                        Kind::Release => crate::release_watch::deputy_brief(r, &self.home),
453                        Kind::Triage => crate::triage::deputy_brief(
454                            r,
455                            &crate::queue::Queue::at(self.home.join("queue")),
456                        ),
457                        Kind::Generic => generic_brief(r, &r.actions, &self.home),
458                    }));
459                }
460                if kind == Kind::Conduct && r.cwd.is_none() {
461                    r.cwd = Some(repo.clone());
462                }
463                if r.answer_timeout == 0 {
464                    r.answer_timeout = match kind {
465                        Kind::Conduct => default_timeout,
466                        Kind::Land => timeout,
467                        Kind::Release | Kind::Triage | Kind::Generic => default_timeout,
468                    };
469                }
470                Ok(())
471            })
472            .map(|(r, ())| r)
473            .map_err(|e| tracing::warn!("question {}: cannot attach a deputy: {e:#}", q.short()))
474            .ok()
475    }
476
477    fn ready(&self, q: Question, now: Timestamp) -> Option<Question> {
478        if (self.halt)() || !q.status.open() || crate::task_chat::routes(&q, &self.home) {
479            return None;
480        }
481        let kind = kind_of(&q)?;
482        let needs_cwd = kind == Kind::Conduct && q.cwd.is_none();
483        let q = if q.deputy.is_none() || needs_cwd || q.answer_timeout == 0 {
484            self.attach(&q, kind)?
485        } else {
486            q
487        };
488        let deputy = q.deputy.as_ref()?;
489        if self
490            .store
491            .read_lease(&q.id)
492            .is_some_and(|lease| lease.fresh(now))
493            || deputy.starts >= MAX_STARTS
494            || !restart_in_time(&q, self.default_timeout(), now)
495            || !can_start(self.cfg.as_ref(), &deputy.agent)
496            || !start_due(&self.store, &q.id, now)
497        {
498            return None;
499        }
500        Some(q)
501    }
502
503    /// Start independent deputy invocations up to the global live reservation
504    /// limit. Return after launch, never after a long question conversation.
505    pub async fn dispatch(&self, now: Timestamp) -> Result<()> {
506        let Some(_dispatch) = crate::worker::try_lease(&self.home, "deputy-dispatch")? else {
507            return Ok(());
508        };
509        let jobs = crate::worker::active(&self.home)?;
510        let mut occupied: HashSet<String> = jobs
511            .iter()
512            .filter(|job| job.kind == "deputy")
513            .map(|job| {
514                job.key
515                    .strip_prefix("deputy:")
516                    .unwrap_or(&job.id)
517                    .to_owned()
518            })
519            .collect();
520        let questions = self.store.list();
521        // Keep first-migration/CLI-owned deputy leases in the same capacity
522        // accounting, without double-counting a lease owned by a known job.
523        occupied.extend(
524            questions
525                .iter()
526                .filter(|q| {
527                    self.store
528                        .read_lease(&q.id)
529                        .is_some_and(|lease| lease.kind == WaiterKind::Deputy && lease.fresh(now))
530                })
531                .map(|q| q.id.clone()),
532        );
533        for q in questions {
534            if occupied.len() >= self.max || (self.halt)() {
535                break;
536            }
537            if occupied.contains(&q.id) {
538                continue;
539            }
540            let Some(q) = self.ready(q, now) else {
541                continue;
542            };
543            let request = WorkerRequest {
544                question: q.id.clone(),
545                cfg: self.cfg.clone(),
546                fallback_repo: self.fallback_repo.clone(),
547            };
548            match crate::worker::spawn(
549                &self.home,
550                "deputy",
551                &format!("deputy:{}", q.id),
552                serde_json::to_value(request)?,
553            )
554            .await
555            {
556                Ok(_) => {
557                    occupied.insert(q.id);
558                }
559                Err(error) => tracing::warn!("question {} deputy worker: {error:#}", q.short()),
560            }
561        }
562        Ok(())
563    }
564
565    /// In-process executor retained for direct runner callers and tests.
566    pub fn tick(&mut self, now: Timestamp) {
567        self.reap();
568        for q in self.store.list() {
569            let Some(q) = self.ready(q, now) else {
570                continue;
571            };
572            if self.inflight.contains(&q.id)
573                || self.inflight.len() >= self.max
574                || matches!(self.memo.get(&q.id), Some(until) if Instant::now() < *until)
575            {
576                continue;
577            }
578            self.memo
579                .insert(q.id.clone(), Instant::now() + RESTART_AFTER);
580            self.inflight.insert(q.id.clone());
581            let job = Job {
582                store: self.store.clone(),
583                home: self.home.clone(),
584                cfg: self.cfg.clone(),
585                fallback_repo: self.fallback_repo.clone(),
586                halt: Arc::clone(&self.halt),
587            };
588            let id = q.id.clone();
589            self.tasks.spawn(async move {
590                if let Err(error) = job.turn(&id).await {
591                    tracing::warn!(
592                        "deputy for question {}: {error:#}",
593                        crate::ask::short_id(&id)
594                    );
595                }
596                id
597            });
598        }
599    }
600
601    /// Wait out invocations started by the in-process runner. The detached
602    /// service dispatcher never uses this path.
603    pub async fn drain(&mut self) {
604        while let Some(done) = self.tasks.join_next().await {
605            if let Ok(id) = done {
606                self.inflight.remove(&id);
607            }
608        }
609        self.inflight.clear();
610    }
611}
612
613#[derive(serde::Serialize, serde::Deserialize)]
614struct WorkerRequest {
615    question: String,
616    cfg: Option<Config>,
617    fallback_repo: PathBuf,
618}
619
620/// Hidden detached entry point for exactly one existing question/seat.
621pub async fn run_worker(home: &Path, payload: serde_json::Value) -> Result<()> {
622    let request: WorkerRequest = serde_json::from_value(payload)?;
623    Job {
624        store: Questions::at(home.join("questions")),
625        home: home.to_path_buf(),
626        cfg: request.cfg,
627        fallback_repo: request.fallback_repo,
628        halt: Arc::new(|| false),
629    }
630    .turn(&request.question)
631    .await
632}
633
634/// Everything one deputy turn needs, owned so it can run detached.
635struct Job {
636    store: Questions,
637    home: PathBuf,
638    cfg: Option<Config>,
639    fallback_repo: PathBuf,
640    halt: Halt,
641}
642
643impl Job {
644    /// Run one invocation, beating the lease; `None` when the daemon is
645    /// parking and the turn was dropped.
646    async fn drive(
647        &self,
648        spec: &crate::config::AgentSpec,
649        seat: &mut SeatState,
650        inv: &Invocation<'_>,
651        id: &str,
652    ) -> Option<Result<agent::AgentOutput>> {
653        let fut = agent::invoke(spec, seat, inv);
654        tokio::pin!(fut);
655        let mut beat = tokio::time::interval(Duration::from_secs(1));
656        let mut beats = 0u32;
657        loop {
658            tokio::select! {
659                r = &mut fut => break Some(r),
660                _ = beat.tick() => {
661                    if (self.halt)() {
662                        break None;
663                    }
664                    beats += 1;
665                    if beats % 20 == 0 {
666                        self.store.beat(id, WaiterKind::Deputy);
667                    }
668                }
669            }
670        }
671    }
672
673    /// The question settled or expired during the handover: release it without
674    /// starting the long turn. The start stays counted - the handover ran.
675    fn park_quietly(&self, id: &str) {
676        let _ = self.store.update(id, |r| {
677            r.waiter = None;
678            Ok(())
679        });
680        self.store.drop_lease(id);
681    }
682
683    /// Parking: the turn is dropped and the start refunded. The seat stays as
684    /// last persisted - after the handover turn, resumable.
685    fn park(&self, id: &str) {
686        let _ = self.store.update(id, |r| {
687            if let Some(d) = r.deputy.as_mut() {
688                d.starts = d.starts.saturating_sub(1);
689            }
690            r.waiter = None;
691            Ok(())
692        });
693        self.store.drop_lease(id);
694    }
695
696    async fn turn(&self, id: &str) -> Result<()> {
697        let claim = self.store.root().join(format!("{id}.deputy-claim"));
698        // The claim only covers the decision to start and the write that records
699        // it, so a daemon that dies holding it blocks a restart for a minute,
700        // not for a turn's length. The lease guards the turn itself.
701        if !crate::waiter::take_claim(&claim, CLAIM_STALE) {
702            return Ok(());
703        }
704        let release = crate::waiter::Release(claim);
705
706        // Decided again under the claim, on the record as it is now.
707        let q = self.store.get(id)?;
708        let now = Timestamp::now();
709        let Some(dep) = q.deputy.clone() else {
710            return Ok(());
711        };
712        if !q.status.open()
713            || crate::task_chat::routes(&q, &self.home)
714            || dep.starts >= MAX_STARTS
715            || self.store.read_lease(&q.id).is_some_and(|l| l.fresh(now))
716            || !start_due(&self.store, &q.id, now)
717        {
718            return Ok(());
719        }
720        let cfg = self
721            .cfg
722            .as_ref()
723            .context("the deputy's configuration is not available")?;
724        let spec = match cfg.agent(&dep.agent) {
725            Ok(s) if !dep.agent.is_empty() => s.clone(),
726            _ => {
727                cfg.resolve_roles()
728                    .context("resolving the deputy's agent")?
729                    .conductor
730            }
731        };
732        // The same CLI conversation when it can be resumed; otherwise a fresh
733        // seat - and either way the prompt carries the whole context.
734        let key = crate::ask::deputy_seat_key(&q.id);
735        let (mut seat, resumed) = match dep.seat.clone() {
736            Some(s)
737                if s.agent == spec.id && agent::has_session(spec.kind, &s, cfg.graph.sessions) =>
738            {
739                (s, true)
740            }
741            _ => (SeatState::new(&key, &spec.id, crate::rng::entropy()), false),
742        };
743        // A release-watch question has no `cwd`; its watch record names the
744        // checkout the pull request belongs to.
745        let recorded = q.cwd.clone().or_else(|| match kind_of(&q) {
746            Some(Kind::Release) => {
747                crate::release_watch::state_for_question(&self.home, &q.id).map(|st| st.repo)
748            }
749            // These record a task id in `run` (or a run id for a divergence's
750            // sibling kinds): the task's repository, when it can be found.
751            Some(Kind::Triage | Kind::Generic) => crate::queue::Queue::at(self.home.join("queue"))
752                .get(&q.run)
753                .ok()
754                .map(|t| t.repo.to_string_lossy().into_owned())
755                .filter(|r| !r.is_empty()),
756            _ => None,
757        });
758        let cwd = recorded
759            .as_deref()
760            .map(PathBuf::from)
761            .filter(|p| p.is_dir())
762            .unwrap_or_else(|| self.fallback_repo.clone());
763
764        // A settle's note is part of what the seat said, so a resumed seat
765        // reads its own report back.
766        let bodies: Vec<String> = q
767            .thread
768            .iter()
769            .map(|t| match &t.note {
770                Some(n) => format!("{}\n(note: {n})", t.body),
771                None => t.body.clone(),
772            })
773            .collect();
774        let thread: Vec<(&str, &str)> = q
775            .thread
776            .iter()
777            .zip(&bodies)
778            .map(|(t, body)| {
779                (
780                    if t.who == Who::Operator {
781                        "operator"
782                    } else {
783                        "agent"
784                    },
785                    body.as_str(),
786                )
787            })
788            .collect();
789        let read = &thread[..q.delivered_turns.min(thread.len())];
790        let unread = q.unread_from_owner();
791        let snapshot = q.thread.len();
792        let current_brief = format!(
793            "{}\n\n{}\n\n{}",
794            dep.brief,
795            crate::choice_effect::brief(&q),
796            crate::queue::dependency_facts(
797                &crate::queue::Queue::at(self.home.join("queue")),
798                &self.store
799            )
800        );
801        let body = prompt::deputy(&prompt::DeputyPrompt {
802            id: &q.id,
803            summary: &q.summary,
804            detail: &q.detail,
805            brief: &current_brief,
806            choices: &q.choices,
807            thread: read,
808            unread: unread.as_deref(),
809            resumed,
810            handover: false,
811            kind: kind_of(&q).unwrap_or(Kind::Conduct),
812            language: &cfg.graph.language,
813        });
814
815        // Recorded before the turn starts, so a daemon that dies mid-turn
816        // leaves a start counted and the seat on the record.
817        let next_start = next_start_path(&self.store, &q.id);
818        std::fs::create_dir_all(next_start.parent().expect("deputy artifacts parent"))?;
819        std::fs::write(
820            next_start,
821            now.as_second()
822                .saturating_add(RESTART_AFTER.as_secs() as i64)
823                .to_string(),
824        )?;
825        self.store.beat(&q.id, WaiterKind::Deputy);
826        let starts = dep.starts + 1;
827        let first = seat.clone();
828        self.store.update(&q.id, |r| {
829            if let Some(d) = r.deputy.as_mut() {
830                d.agent = spec.id.clone();
831                d.seat = Some(first);
832                d.starts = starts;
833            }
834            r.waiter = Some(Note {
835                kind: WaiterKind::Deputy,
836                since: now,
837            });
838            Ok(())
839        })?;
840        drop(release);
841        tracing::info!(
842            "question {}: deputy seat {} {} (start {starts}/{MAX_STARTS})",
843            q.short(),
844            seat.key,
845            if resumed { "resuming" } else { "starting" }
846        );
847
848        let left = (deadline(&q, cfg.graph.answer_timeout) - now.as_second()).max(0) as u64;
849        let artifacts = self.store.root().join(format!("{}.deputy", q.id));
850        let cache_dir = cfg.cache_dir();
851        // `magi ask` writes the question record and its lock, which a read-only
852        // sandbox refuses, so the seat cannot be read-only. It is told never to
853        // edit anything; the daemon, not the deputy, applies outcomes.
854        let allow_write = true;
855        // The question store is outside the repository, and `magi ask` writes it.
856        // A merge approval's deputy also files follow-up tasks with `magi task
857        // add`, which writes the queue; that is the one other place it writes.
858        let writable = [self.store.root().to_path_buf(), self.home.join("queue")];
859        macro_rules! invocation {
860            ($prompt:expr, $stem:expr, $timeout:expr) => {
861                Invocation {
862                    scope: crate::agent::InvocationScope::Repository,
863                    coordination: cfg.coordination.as_ref(),
864                    fetched_base: None,
865                    cwd: &cwd,
866                    prompt: $prompt,
867                    timeout: $timeout,
868                    allow_write,
869                    unsandboxed: false,
870                    sessions: cfg.graph.sessions,
871                    artifacts: &artifacts,
872                    stem: $stem,
873                    // The question's own run key (the task id) is what lets this
874                    // seat's `magi ask` pass the ownership check on a conductor
875                    // question.
876                    run: &q.run,
877                    node: NODE,
878                    cache_dir: cache_dir.as_deref(),
879                    attachments: &[],
880                    writable: &writable,
881                }
882            };
883        }
884
885        // A fresh seat first takes a short turn that ends: `agent::invoke` only
886        // learns the CLI's session id when it returns, and the real turn blocks
887        // in `magi ask` for hours, so a daemon stopped mid-wait would otherwise
888        // leave nothing to resume. This turn persists the seat before the wait.
889        let mut early = None;
890        if !resumed && cfg.graph.sessions {
891            let hbody = prompt::deputy(&prompt::DeputyPrompt {
892                id: &q.id,
893                summary: &q.summary,
894                detail: &q.detail,
895                brief: &current_brief,
896                choices: &q.choices,
897                thread: read,
898                unread: None,
899                resumed: false,
900                handover: true,
901                kind: kind_of(&q).unwrap_or(Kind::Conduct),
902                language: &cfg.graph.language,
903            });
904            let hstem = format!("handover-{starts}");
905            // Bounded by the question's own deadline, not only by its own cap: the
906            // lease it beats would otherwise keep an expired question alive.
907            let hlimit = HANDOVER_TIMEOUT.min(Duration::from_secs(left.max(1)));
908            let hinv = invocation!(&hbody, &hstem, hlimit);
909            let Some(done) = self.drive(&spec, &mut seat, &hinv, &q.id).await else {
910                self.park(&q.id);
911                return Ok(());
912            };
913            let kept = seat.clone();
914            self.store.update(&q.id, |r| {
915                if let Some(d) = r.deputy.as_mut() {
916                    d.seat = Some(kept);
917                }
918                Ok(())
919            })?;
920            if !matches!(&done, Ok(o) if o.usable()) {
921                early = Some(done);
922            }
923        }
924
925        // The handover may have been slow: look at the question again, and
926        // measure the long turn from what is left now.
927        let left = if early.is_none() && !resumed && cfg.graph.sessions {
928            let now = Timestamp::now();
929            let again = self.store.get(&q.id)?;
930            let left = (deadline(&again, cfg.graph.answer_timeout) - now.as_second()).max(0) as u64;
931            if !again.status.open() || (left == 0 && again.unread_from_owner().is_none()) {
932                self.park_quietly(&q.id);
933                return Ok(());
934            }
935            left
936        } else {
937            left
938        };
939        let out = match early {
940            Some(done) => done,
941            None => {
942                let stem = format!("turn-{starts}");
943                let timeout = Duration::from_secs(left.max(60) + SLACK_SECS);
944                let inv = invocation!(&body, &stem, timeout);
945                match self.drive(&spec, &mut seat, &inv, &q.id).await {
946                    Some(out) => out,
947                    None => {
948                        self.park(&q.id);
949                        return Ok(());
950                    }
951                }
952            }
953        };
954
955        let (text, why) = match out {
956            Ok(o) if o.usable() => (Some(o.text.trim().to_owned()), None),
957            Ok(o) if o.timed_out => (None, Some("its turn timed out".to_owned())),
958            Ok(o) if o.quota_exhausted() => (None, Some("the agent is out of quota".to_owned())),
959            Ok(o) => (
960                None,
961                Some(format!(
962                    "its turn failed ({})",
963                    notices::exit_status(o.exit_code)
964                )),
965            ),
966            Err(e) => (None, Some(format!("the agent could not be started: {e:#}"))),
967        };
968        let kept = seat.clone();
969        self.store.update(&q.id, |r| {
970            if let Some(d) = r.deputy.as_mut() {
971                d.seat = Some(kept);
972            }
973            r.waiter = None;
974            // Retire ownership under the same lock used by text admission.
975            self.store.drop_lease(&q.id);
976            // The owner's words were in the prompt. An agent that read them but
977            // answered in prose instead of `magi ask --thread` still spoke;
978            // keep it on the record so the owner reads it.
979            if let Some(text) = text
980                && unread.is_some()
981                && !text.is_empty()
982                && r.status.open()
983                && r.thread.len() == snapshot
984            {
985                r.delivered_turns = r.delivered_turns.max(snapshot);
986                let choices = r.choices.clone();
987                r.reply(text, choices)?;
988            }
989            Ok(())
990        })?;
991        if let Some(why) = why {
992            tracing::warn!("question {}: the deputy ended: {why}", q.short());
993            notices::raise_in(
994                &self.home,
995                Notice::warn(
996                    &format!("deputy-turn:{}", q.id),
997                    format!(
998                        "Question {} \"{}\": its follow-up agent stopped ({why}).",
999                        q.short(),
1000                        q.summary
1001                    ),
1002                )
1003                .link(if q.run_names_task() {
1004                    notices::Link::Task { id: q.run.clone() }
1005                } else {
1006                    notices::Link::Run { id: q.run.clone() }
1007                }),
1008            );
1009        }
1010        Ok(())
1011    }
1012}
1013
1014#[cfg(test)]
1015mod worker_tests {
1016    use super::*;
1017
1018    #[test]
1019    fn restart_cooldown_survives_a_new_runner() {
1020        let home = tempfile::tempdir().expect("home");
1021        let questions = Questions::at(home.path().join("questions"));
1022        let path = next_start_path(&questions, "question");
1023        std::fs::create_dir_all(path.parent().expect("parent")).expect("artifacts");
1024        std::fs::write(&path, "130").expect("next start");
1025        let reopened = Questions::at(home.path().join("questions"));
1026        assert!(!start_due(
1027            &reopened,
1028            "question",
1029            Timestamp::from_second(129).unwrap()
1030        ));
1031        assert!(start_due(
1032            &reopened,
1033            "question",
1034            Timestamp::from_second(130).unwrap()
1035        ));
1036    }
1037}