Skip to main content

magi/
deputy.rs

1//! Deputies: the seat that waits on a conductor's question, so the conductor
2//! never has to - and on land's merge approval, whose free-text reply used to
3//! reach nobody either.
4//!
5//! [`crate::conduct`] files a question for the operator and moves on; it is
6//! non-blocking by construction, because one task's question must not park the
7//! whole polling loop. The price used to be that nothing at all was waiting on
8//! it: the owner's free-text reply landed in a thread no agent read, while the
9//! phone said "waiting for the agent". Question f1dc on task 684d sat open
10//! forever on a "setup done".
11//!
12//! A deputy is the answer. For each open conductor question, `magi serve`
13//! runs one short-lived agent seat that is handed what the conductor knew
14//! ([`brief`]: the task, why it asked, what each option leads to), blocks on
15//! the question with `magi ask --wait`, answers back in the thread with
16//! `magi ask --thread`, and - when the owner's words pick an option - records
17//! that with `magi ask --settle`. Applying the outcome is not
18//! the deputy's job: an answered conductor question goes through the daemon's
19//! existing `resolve_blockers` / action path, exactly as one tapped on the
20//! phone does.
21//!
22//! # Invariants
23//!
24//! - **One seat per question, keyed by seat name.** `deputy-<question id>`,
25//!   never the conductor's shared seat (`conduct/seat.json`, overwritten by
26//!   every cycle) and never an agent id.
27//! - **Persisted on the question** ([`crate::ask::Deputy`]), always through
28//!   [`Questions::update`]: the owner's say and answer write the same file. A
29//!   restarted daemon resumes the seat only when [`agent::has_session`] says
30//!   it can; otherwise it starts a fresh seat and re-sends the whole context,
31//!   never assuming memory.
32//! - **Bounded.** At most `daemon.max_deputies` at once; at most
33//!   [`MAX_STARTS`] starts per question, and a restart does not reset that. A
34//!   deputy never outlives the question's `answer_timeout`: the waiter retires
35//!   the question at its deadline ([`crate::waiter`]) and the task goes to a
36//!   machine hold (`daemon::resolve_blockers`).
37//! - **Never a second agent on one question.** A fresh lease or the claim file
38//!   means somebody is already on it.
39//! - **A merge approval is served too, but stays land's.** Its deputy has no
40//!   `cwd` (so the waiter never touches it), its deadline is `asked_at +
41//!   answer_timeout` and never moves on a reply ([`deadline`]), and the only
42//!   thing that retires it is `daemon::land_resume_state`. Silence is a hold;
43//!   a settled `merge` lets land merge the head it saved, nothing newer.
44//! - **A release-watch question is served too, and stays the watcher's.** The
45//!   escalation, local-mode approval and failed-release questions
46//!   ([`crate::release_watch`], node `release-bump`, seat `release-watch`) have
47//!   no asker, so a say reached nobody. Its deputy has no `cwd` either, a
48//!   fixed `asked_at + answer_timeout` deadline ([`deadline`]), and applies
49//!   nothing: `--settle` records a choice and the watcher applies it on its next
50//!   lap. A deputy never closes or merges the pull request itself.
51//! - **Every open question the owner can say something to has a listener.**
52//!   [`Kind::Triage`] serves the triage questions; [`Kind::Generic`] is the
53//!   fallback for any other question with choices and no `cwd` (the divergence
54//!   question today, a node nobody has written yet tomorrow). It is decided by
55//!   the missing `cwd`, never by node name. Only a conductor question is ever
56//!   given a `cwd` ([`Deputies::attach`]); the other kinds would otherwise be
57//!   resumed and expired by the waiter beside the deputy. A test enumerates
58//!   every node filed without an asker, so a new kind of question cannot ship
59//!   without one.
60//! - **The owner's words are the authority.** The deputy reads what the owner
61//!   said and carries it out: `--settle` records the offered label it read
62//!   from those words, with no quote. Magi checks only that the caller is this
63//!   question's deputy seat, the label is offered and the question is open;
64//!   applying it stays with the daemon or the component that asked.
65
66use std::collections::{BTreeMap, HashMap, HashSet};
67use std::path::{Path, PathBuf};
68use std::sync::Arc;
69use std::time::{Duration, Instant};
70
71use anyhow::{Context as _, Result};
72use jiff::Timestamp;
73use tokio::task::JoinSet;
74
75use crate::agent::{self, Invocation, SeatState};
76use crate::ask::{ChoiceAction, Deputy, Question, Questions, Waiter as Note, WaiterKind, Who};
77use crate::config::Config;
78use crate::notices::{self, Notice};
79use crate::prompt;
80
81/// Node name a deputy runs under (`MAGI_NODE`); `magi ask --settle` accepts
82/// it and nothing else.
83pub const NODE: &str = "deputy";
84
85/// How often the runner looks at the store.
86pub const TICK: Duration = Duration::from_secs(5);
87
88/// Most turns ever started for one question. A deputy that keeps ending
89/// without an answer is a problem for the owner, not something to retry until
90/// the quota is gone.
91pub const MAX_STARTS: u32 = 3;
92
93/// Least gap between two starts for one question, including new owner input.
94/// Persisted independently of a runner so finite service laps cannot repeatedly
95/// bill the same conversation or exhaust its lifetime start quota.
96const RESTART_AFTER: Duration = Duration::from_secs(30);
97
98fn next_start_path(store: &Questions, id: &str) -> PathBuf {
99    store.root().join(format!("{id}.deputy")).join("next-start")
100}
101
102fn start_due(store: &Questions, id: &str, now: Timestamp) -> bool {
103    std::fs::read_to_string(next_start_path(store, id))
104        .ok()
105        .and_then(|text| text.parse::<i64>().ok())
106        .is_none_or(|next| now.as_second() >= next)
107}
108
109/// Added to the question's remaining time for the invocation's wall clock, so
110/// the deputy's own `magi ask --wait` reaches the deadline first.
111const SLACK_SECS: u64 = 120;
112
113/// A claim file older than this was left by a process that died.
114const CLAIM_STALE: Duration = Duration::from_secs(60);
115
116/// How long the context-handover turn may take.
117const HANDOVER_TIMEOUT: Duration = Duration::from_secs(10 * 60);
118
119/// Stops the deputy when true: the daemon is parking.
120pub type Halt = Arc<dyn Fn() -> bool + Send + Sync>;
121
122/// What the conductor knew about one question, for its deputy.
123///
124/// `reason` is the conductor's own one-line reasoning (may be empty). Each
125/// choice is listed with what picking it does: an attached action, or - for
126/// the usual conductor question - that the answer is recorded on the task and
127/// the task is unblocked with it in its instructions.
128pub fn brief(
129    task_id: &str,
130    reason: &str,
131    choices: &[String],
132    actions: &BTreeMap<String, ChoiceAction>,
133) -> String {
134    let reason = reason.trim();
135    let mut s = format!(
136        "Task {task_id} (`magi task show {task_id}`) is blocked on this question. \
137         The conductor asked because: {}\n\n\
138         When the owner answers, the daemon applies the exact attached action \
139         once, after rechecking its preconditions. You only record the choice; \
140         you never execute maintenance or edit repository or run records.",
141        if reason.is_empty() {
142            "(it recorded no reasoning)"
143        } else {
144            reason
145        }
146    );
147    if !choices.is_empty() {
148        s.push_str("\n\nWhat each option does:");
149        for c in choices {
150            match actions.get(c) {
151                Some(a) => s.push_str(&format!("\n- `{c}`: also {}", a.describe())),
152                None => s.push_str(&format!(
153                    "\n- `{c}`: no executable operation; keep held and ask for a supported choice"
154                )),
155            }
156        }
157    }
158    s
159}
160
161/// What a deputy serves: the question kinds it is attached to.
162#[derive(Debug, Clone, Copy, PartialEq, Eq)]
163pub enum Kind {
164    /// A conductor's question ([`crate::conduct::NODE`]).
165    Conduct,
166    /// The merge approval ([`crate::land::APPROVAL_NODE`]).
167    Land,
168    /// A question the release watcher filed ([`crate::release_watch`]).
169    Release,
170    /// A triage question about a held task or a stuck dependency root
171    /// ([`crate::triage::NODE`], [`crate::triage::DEPS_NODE`]).
172    Triage,
173    /// Any other question nobody asked from inside a run (no `cwd`) that offers
174    /// choices: the fallback, so a node nobody has met yet still has a listener.
175    Generic,
176}
177
178/// The kind of question a deputy serves for `q`, `None` for every other.
179///
180/// The fallback is decided by the absence of a `cwd`, never by the node name: a
181/// question filed with `magi ask` inside a run records its `cwd` and has an
182/// asker (and the waiter), and the node a seat asks from is the seat's own name
183/// (a reviewer's `review` is also the divergence question's node). A question
184/// with no choices cannot be settled and is a notice, not something to answer.
185pub fn kind_of(q: &Question) -> Option<Kind> {
186    if q.routed_task.is_some() {
187        return None;
188    }
189    match q.node.as_str() {
190        crate::conduct::NODE => Some(Kind::Conduct),
191        crate::land::APPROVAL_NODE => Some(Kind::Land),
192        // The seat matters: `bump` files choice-less notices on the same node,
193        // which nobody can answer and which must not cost a deputy.
194        crate::bump::NOTICE_NODE if q.seat == "release-watch" => Some(Kind::Release),
195        crate::bump::NOTICE_NODE => None,
196        crate::triage::NODE | crate::triage::DEPS_NODE => Some(Kind::Triage),
197        _ if q.cwd.is_none() && !q.choices.is_empty() => Some(Kind::Generic),
198        _ => None,
199    }
200}
201
202/// What a deputy is told about a question of no known kind: only what the
203/// question itself stored.
204pub fn generic_brief(
205    q: &Question,
206    actions: &BTreeMap<String, ChoiceAction>,
207    home: &Path,
208) -> String {
209    // `Question::run` may also hold a task id; only a run id can be shown.
210    let run = crate::run::is_run_id(&q.run).then_some(q.run.as_str());
211    let knows = if run.is_some() {
212        "You know what the question itself says and what you can learn from \
213         its run by the read-only investigation described below."
214    } else {
215        "You know only what the question itself says."
216    };
217    let mut s = format!(
218        "This question was filed by magi (node `{}`, seat `{}`) with no agent \
219         waiting on it. Magi records the owner's answer and the component that \
220         asked applies it; you apply nothing. {knows} When the owner's words \
221         pick an option, settle it; when you cannot tell which option they \
222         mean, ask them with `--thread`.",
223        q.node, q.seat
224    );
225    if let Some(run) = run {
226        let artifacts = home.join("runs").join(run).join("artifacts");
227        s.push_str(&format!(
228            "\n\nThis question concerns run `{run}`. You may investigate it on \
229             your own, read-only: run `magi show {run}` and read files under \
230             `{}`. When the owner asks about details (for example which part \
231             was wrong), look it up yourself and answer; do not reply that you \
232             cannot see the details, and do not tell the owner to run `magi show` \
233             themselves. Investigating never changes anything: edit, commit, push \
234             and apply nothing. If the run cannot be read, say exactly that and do \
235             not guess at the cause or at what an option does.",
236            artifacts.display()
237        ));
238    }
239    if q.node == crate::github_text::ASK_NODE {
240        s.push_str("\n\n");
241        s.push_str(&crate::github_text::deputy_brief());
242    }
243    if !q.choices.is_empty() {
244        s.push_str("\n\nWhat each option does:");
245        for c in &q.choices {
246            match actions.get(c) {
247                Some(a) => s.push_str(&format!("\n- `{c}`: also {}", a.describe())),
248                None => s.push_str(&format!(
249                    "\n- `{c}`: {}",
250                    crate::choice_effect::describe(q, c)
251                        .unwrap_or_else(|| "unsupported legacy choice; nothing applies it".into())
252                )),
253            }
254        }
255    }
256    s
257}
258
259/// Does `q`'s clock run from `asked_at` and never move on a reply? True for the
260/// questions that something other than the waiter retires: a merge approval
261/// (land) and a release-watch question (the watcher, by silence being a hold).
262pub fn fixed_clock(q: &Question) -> bool {
263    q.node == crate::land::APPROVAL_NODE
264        || (q.node == crate::bump::NOTICE_NODE && q.seat == "release-watch")
265        || q.node == crate::github_text::ASK_NODE
266}
267
268/// Second after which nobody is to be started or kept on `q`.
269///
270/// A conductor question runs from its last activity (`magi ask --thread`
271/// re-arms it, and the waiter retires it). A merge approval never moves: it
272/// runs from `asked_at`, exactly where `daemon::land_resume_state` abandons it,
273/// so a conversation cannot stretch the hold and land stays the only place that
274/// retires one.
275pub fn deadline(q: &Question, default_timeout: u64) -> i64 {
276    let secs = if q.answer_timeout > 0 {
277        q.answer_timeout
278    } else {
279        default_timeout
280    };
281    let from = if fixed_clock(q) {
282        q.asked_at.as_second()
283    } else {
284        q.last_activity()
285    };
286    from.saturating_add(secs as i64)
287}
288
289/// Dispatch and text admission share the fixed-clock and unread-word rules.
290pub fn restart_in_time(q: &Question, default_timeout: u64, now: Timestamp) -> bool {
291    now.as_second() <= deadline(q, default_timeout) || q.unread_from_owner().is_some()
292}
293
294/// Can `magi serve` start a deputy at all under `cfg`? Not when deputies are
295/// switched off (`daemon.max_deputies = 0`) or the config could not be read.
296///
297/// Also false when the agent the deputy would run as cannot be resolved
298/// (`agent` is the deputy's recorded agent, empty when it has none): `turn`
299/// fails on that before it counts a start, so it would never reach
300/// [`MAX_STARTS`].
301pub fn can_start(cfg: Option<&Config>, agent: &str) -> bool {
302    cfg.is_some_and(|c| {
303        c.daemon.max_deputies > 0
304            && ((!agent.is_empty() && c.agent(agent).is_ok()) || c.resolve_roles().is_ok())
305    })
306}
307
308/// The agent a question's deputy records, empty when it has none yet.
309pub fn agent_of(q: &Question) -> &str {
310    q.deputy.as_ref().map_or("", |d| d.agent.as_str())
311}
312
313/// Has this question's deputy run out of starts, or can none ever start, with
314/// the deadline gone?
315///
316/// Then nothing will ever read an unread say, and the waiter must retire the
317/// question anyway instead of deferring to a deputy that no longer starts.
318/// `startable` is [`can_start`]; a fresh lease is the caller's to check.
319pub fn exhausted_past_deadline(
320    q: &Question,
321    startable: bool,
322    default_timeout: u64,
323    now: Timestamp,
324) -> bool {
325    q.status.open()
326        && q.deputy
327            .as_ref()
328            .is_some_and(|d| d.starts >= MAX_STARTS || !startable)
329        && now.as_second() > deadline(q, default_timeout)
330}
331
332/// Question-service dispatcher; each invocation owns an independent worker.
333pub struct Deputies {
334    store: Questions,
335    home: PathBuf,
336    cfg: Option<Config>,
337    /// Working directory for a question that recorded none.
338    fallback_repo: PathBuf,
339    max: usize,
340    halt: Halt,
341    tasks: JoinSet<String>,
342    inflight: HashSet<String>,
343    /// Earliest next start per question.
344    memo: HashMap<String, Instant>,
345}
346
347impl Deputies {
348    /// A runner over `store`, with at most `max` deputies at once.
349    pub fn new(
350        store: Questions,
351        home: PathBuf,
352        cfg: Option<Config>,
353        fallback_repo: PathBuf,
354        max: usize,
355        halt: Halt,
356    ) -> Self {
357        Self {
358            store,
359            home,
360            cfg,
361            fallback_repo,
362            max,
363            halt,
364            tasks: JoinSet::new(),
365            inflight: HashSet::new(),
366            memo: HashMap::new(),
367        }
368    }
369
370    /// Admission check for a finite tick; an idle store needs no child process.
371    pub fn has_work(&self, now: Timestamp) -> bool {
372        self.max > 0
373            && self.store.list().iter().any(|q| {
374                kind_of(q).is_some()
375                    && !crate::task_chat::routes(q, &self.home)
376                    && q.status.open()
377                    && q.deputy
378                        .as_ref()
379                        .is_none_or(|deputy| deputy.starts < MAX_STARTS)
380                    && self
381                        .store
382                        .read_lease(&q.id)
383                        .is_none_or(|lease| !lease.fresh(now))
384                    && can_start(self.cfg.as_ref(), agent_of(q))
385                    && restart_in_time(q, self.default_timeout(), now)
386                    && start_due(&self.store, &q.id, now)
387            })
388    }
389
390    fn default_timeout(&self) -> u64 {
391        self.cfg
392            .as_ref()
393            .map_or(Config::default().graph.answer_timeout, |c| {
394                c.graph.answer_timeout
395            })
396    }
397
398    fn reap(&mut self) {
399        while let Some(done) = self.tasks.try_join_next() {
400            if let Ok(id) = done {
401                self.inflight.remove(&id);
402            }
403        }
404        if self.tasks.is_empty() {
405            self.inflight.clear();
406        }
407    }
408
409    /// Give a question the record a deputy needs: the deputy itself (with a
410    /// brief rebuilt from what the question stored, when it was filed before
411    /// deputies existed - a lost reason is not invented) and the deadline.
412    ///
413    /// A conductor question also gets the working directory the waiter and
414    /// `magi ask --wait` use. Every other kind never does: `cwd` is what makes a
415    /// question the waiter's (it would resume and expire it beside the deputy),
416    /// and a release-watch question, a merge approval, a triage question and a
417    /// fallback one have no asker to resume.
418    fn attach(&self, q: &Question, kind: Kind) -> Option<Question> {
419        let default_timeout = self.default_timeout();
420        let repo = self.fallback_repo.to_string_lossy().into_owned();
421        let state = match kind {
422            Kind::Land => crate::run::RunState::load(&q.run).ok(),
423            Kind::Conduct | Kind::Release | Kind::Triage | Kind::Generic => None,
424        };
425        // The deadline land itself enforces for this run.
426        let timeout = state
427            .as_ref()
428            .map_or(default_timeout, |s| s.config.graph.answer_timeout);
429        self.store
430            .update(&q.id, |r| {
431                if r.deputy.is_none() {
432                    r.deputy = Some(Deputy::new(brief_for(&self.home, r, kind, state.as_ref())));
433                }
434                if kind == Kind::Conduct && r.cwd.is_none() {
435                    r.cwd = Some(repo.clone());
436                }
437                if r.answer_timeout == 0 {
438                    r.answer_timeout = match kind {
439                        Kind::Conduct => default_timeout,
440                        Kind::Land => timeout,
441                        Kind::Release | Kind::Triage | Kind::Generic => default_timeout,
442                    };
443                }
444                Ok(())
445            })
446            .map(|(r, ())| r)
447            .map_err(|e| tracing::warn!("question {}: cannot attach a deputy: {e:#}", q.short()))
448            .ok()
449    }
450
451    fn ready(&self, q: Question, now: Timestamp) -> Option<Question> {
452        if (self.halt)() || !q.status.open() || crate::task_chat::routes(&q, &self.home) {
453            return None;
454        }
455        let kind = kind_of(&q)?;
456        let needs_cwd = kind == Kind::Conduct && q.cwd.is_none();
457        let q = if q.deputy.is_none() || needs_cwd || q.answer_timeout == 0 {
458            self.attach(&q, kind)?
459        } else {
460            q
461        };
462        let deputy = q.deputy.as_ref()?;
463        if self
464            .store
465            .read_lease(&q.id)
466            .is_some_and(|lease| lease.fresh(now))
467            || deputy.starts >= MAX_STARTS
468            || !restart_in_time(&q, self.default_timeout(), now)
469            || !can_start(self.cfg.as_ref(), &deputy.agent)
470            || !start_due(&self.store, &q.id, now)
471        {
472            return None;
473        }
474        Some(q)
475    }
476
477    /// Start independent deputy invocations up to the global live reservation
478    /// limit. Return after launch, never after a long question conversation.
479    pub async fn dispatch(&self, now: Timestamp) -> Result<()> {
480        let Some(_dispatch) = crate::worker::try_lease(&self.home, "deputy-dispatch")? else {
481            return Ok(());
482        };
483        let jobs = crate::worker::active(&self.home)?;
484        let mut occupied: HashSet<String> = jobs
485            .iter()
486            .filter(|job| job.kind == "deputy")
487            .map(|job| {
488                job.key
489                    .strip_prefix("deputy:")
490                    .unwrap_or(&job.id)
491                    .to_owned()
492            })
493            .collect();
494        let questions = self.store.list();
495        // Keep first-migration/CLI-owned deputy leases in the same capacity
496        // accounting, without double-counting a lease owned by a known job.
497        occupied.extend(
498            questions
499                .iter()
500                .filter(|q| {
501                    self.store
502                        .read_lease(&q.id)
503                        .is_some_and(|lease| lease.kind == WaiterKind::Deputy && lease.fresh(now))
504                })
505                .map(|q| q.id.clone()),
506        );
507        for q in questions {
508            if occupied.len() >= self.max || (self.halt)() {
509                break;
510            }
511            if occupied.contains(&q.id) {
512                continue;
513            }
514            let Some(q) = self.ready(q, now) else {
515                continue;
516            };
517            let request = WorkerRequest {
518                question: q.id.clone(),
519                cfg: self.cfg.clone(),
520                fallback_repo: self.fallback_repo.clone(),
521            };
522            match crate::worker::spawn(
523                &self.home,
524                "deputy",
525                &format!("deputy:{}", q.id),
526                serde_json::to_value(request)?,
527            )
528            .await
529            {
530                Ok(_) => {
531                    occupied.insert(q.id);
532                }
533                Err(error) => tracing::warn!("question {} deputy worker: {error:#}", q.short()),
534            }
535        }
536        Ok(())
537    }
538
539    /// In-process executor retained for direct runner callers and tests.
540    pub fn tick(&mut self, now: Timestamp) {
541        self.reap();
542        for q in self.store.list() {
543            let Some(q) = self.ready(q, now) else {
544                continue;
545            };
546            if self.inflight.contains(&q.id)
547                || self.inflight.len() >= self.max
548                || matches!(self.memo.get(&q.id), Some(until) if Instant::now() < *until)
549            {
550                continue;
551            }
552            self.memo
553                .insert(q.id.clone(), Instant::now() + RESTART_AFTER);
554            self.inflight.insert(q.id.clone());
555            let job = Job {
556                store: self.store.clone(),
557                home: self.home.clone(),
558                cfg: self.cfg.clone(),
559                fallback_repo: self.fallback_repo.clone(),
560                halt: Arc::clone(&self.halt),
561            };
562            let id = q.id.clone();
563            self.tasks.spawn(async move {
564                if let Err(error) = job.turn(&id).await {
565                    tracing::warn!(
566                        "deputy for question {}: {error:#}",
567                        crate::ask::short_id(&id)
568                    );
569                }
570                id
571            });
572        }
573    }
574
575    /// Wait out invocations started by the in-process runner. The detached
576    /// service dispatcher never uses this path.
577    pub async fn drain(&mut self) {
578        while let Some(done) = self.tasks.join_next().await {
579            if let Ok(id) = done {
580                self.inflight.remove(&id);
581            }
582        }
583        self.inflight.clear();
584    }
585}
586
587/// The brief for a `kind` question, from what the question stored and the
588/// current state (`state` is the run record of a merge approval).
589fn brief_for(
590    home: &Path,
591    q: &Question,
592    kind: Kind,
593    state: Option<&crate::run::RunState>,
594) -> String {
595    match kind {
596        Kind::Conduct => brief(&q.run, &q.detail, &q.choices, &q.actions),
597        Kind::Land => crate::land::deputy_brief(q, state),
598        Kind::Release => crate::release_watch::deputy_brief(q, home),
599        Kind::Triage => {
600            crate::triage::deputy_brief(q, &crate::queue::Queue::at(home.join("queue")))
601        }
602        Kind::Generic => generic_brief(q, &q.actions, home),
603    }
604}
605
606#[derive(serde::Serialize, serde::Deserialize)]
607struct WorkerRequest {
608    question: String,
609    cfg: Option<Config>,
610    fallback_repo: PathBuf,
611}
612
613/// Hidden detached entry point for exactly one existing question/seat.
614pub async fn run_worker(home: &Path, payload: serde_json::Value) -> Result<()> {
615    let request: WorkerRequest = serde_json::from_value(payload)?;
616    Job {
617        store: Questions::at(home.join("questions")),
618        home: home.to_path_buf(),
619        cfg: request.cfg,
620        fallback_repo: request.fallback_repo,
621        halt: Arc::new(|| false),
622    }
623    .turn(&request.question)
624    .await
625}
626
627/// Everything one deputy turn needs, owned so it can run detached.
628struct Job {
629    store: Questions,
630    home: PathBuf,
631    cfg: Option<Config>,
632    fallback_repo: PathBuf,
633    halt: Halt,
634}
635
636impl Job {
637    /// Run one invocation, beating the lease; `None` when the daemon is
638    /// parking and the turn was dropped.
639    async fn drive(
640        &self,
641        spec: &crate::config::AgentSpec,
642        seat: &mut SeatState,
643        inv: &Invocation<'_>,
644        id: &str,
645    ) -> Option<Result<agent::AgentOutput>> {
646        let fut = agent::invoke(spec, seat, inv);
647        tokio::pin!(fut);
648        let mut beat = tokio::time::interval(Duration::from_secs(1));
649        let mut beats = 0u32;
650        loop {
651            tokio::select! {
652                r = &mut fut => break Some(r),
653                _ = beat.tick() => {
654                    if (self.halt)() {
655                        break None;
656                    }
657                    beats += 1;
658                    if beats % 20 == 0 {
659                        self.store.beat(id, WaiterKind::Deputy);
660                    }
661                }
662            }
663        }
664    }
665
666    /// The question settled or expired during the handover: release it without
667    /// starting the long turn. The start stays counted - the handover ran.
668    fn park_quietly(&self, id: &str) {
669        let _ = self.store.update(id, |r| {
670            r.waiter = None;
671            Ok(())
672        });
673        self.store.drop_lease(id);
674    }
675
676    /// Parking: the turn is dropped and the start refunded. The seat stays as
677    /// last persisted - after the handover turn, resumable.
678    fn park(&self, id: &str) {
679        let _ = self.store.update(id, |r| {
680            if let Some(d) = r.deputy.as_mut() {
681                d.starts = d.starts.saturating_sub(1);
682            }
683            r.waiter = None;
684            Ok(())
685        });
686        self.store.drop_lease(id);
687    }
688
689    async fn turn(&self, id: &str) -> Result<()> {
690        let claim = self.store.root().join(format!("{id}.deputy-claim"));
691        // The claim only covers the decision to start and the write that records
692        // it, so a daemon that dies holding it blocks a restart for a minute,
693        // not for a turn's length. The lease guards the turn itself.
694        if !crate::waiter::take_claim(&claim, CLAIM_STALE) {
695            return Ok(());
696        }
697        let release = crate::waiter::Release(claim);
698
699        // Decided again under the claim, on the record as it is now.
700        let q = self.store.get(id)?;
701        let now = Timestamp::now();
702        let Some(dep) = q.deputy.clone() else {
703            return Ok(());
704        };
705        if !q.status.open()
706            || crate::task_chat::routes(&q, &self.home)
707            || dep.starts >= MAX_STARTS
708            || self.store.read_lease(&q.id).is_some_and(|l| l.fresh(now))
709            || !start_due(&self.store, &q.id, now)
710        {
711            return Ok(());
712        }
713        let cfg = self
714            .cfg
715            .as_ref()
716            .context("the deputy's configuration is not available")?;
717        let spec = match cfg.agent(&dep.agent) {
718            Ok(s) if !dep.agent.is_empty() => s.clone(),
719            _ => {
720                cfg.resolve_roles()
721                    .context("resolving the deputy's agent")?
722                    .conductor
723            }
724        };
725        // The same CLI conversation when it can be resumed; otherwise a fresh
726        // seat - and either way the prompt carries the whole context.
727        let key = crate::ask::deputy_seat_key(&q.id);
728        let (mut seat, resumed) = match dep.seat.clone() {
729            Some(s)
730                if s.agent == spec.id && agent::has_session(spec.kind, &s, cfg.graph.sessions) =>
731            {
732                (s, true)
733            }
734            _ => (SeatState::new(&key, &spec.id, crate::rng::entropy()), false),
735        };
736        // A release-watch question has no `cwd`; its watch record names the
737        // checkout the pull request belongs to.
738        let recorded = q.cwd.clone().or_else(|| match kind_of(&q) {
739            Some(Kind::Release) => {
740                crate::release_watch::state_for_question(&self.home, &q.id).map(|st| st.repo)
741            }
742            // These record a task id in `run` (or a run id for a divergence's
743            // sibling kinds): the task's repository, when it can be found.
744            Some(Kind::Triage | Kind::Generic) => crate::queue::Queue::at(self.home.join("queue"))
745                .get(&q.run)
746                .ok()
747                .map(|t| t.repo.to_string_lossy().into_owned())
748                .filter(|r| !r.is_empty()),
749            _ => None,
750        });
751        let cwd = recorded
752            .as_deref()
753            .map(PathBuf::from)
754            .filter(|p| p.is_dir())
755            .unwrap_or_else(|| self.fallback_repo.clone());
756
757        // A settle's note is part of what the seat said, so a resumed seat
758        // reads its own report back.
759        let bodies: Vec<String> = q
760            .thread
761            .iter()
762            .map(|t| match &t.note {
763                Some(n) => format!("{}\n(note: {n})", t.body),
764                None => t.body.clone(),
765            })
766            .collect();
767        let thread: Vec<(&str, &str)> = q
768            .thread
769            .iter()
770            .zip(&bodies)
771            .map(|(t, body)| {
772                (
773                    if t.who == Who::Operator {
774                        "operator"
775                    } else {
776                        "agent"
777                    },
778                    body.as_str(),
779                )
780            })
781            .collect();
782        let read = &thread[..q.delivered_turns.min(thread.len())];
783        let unread = q.unread_from_owner();
784        let snapshot = q.thread.len();
785        // Only a conductor question's stored brief reaches the seat: it holds
786        // the conductor's own reasoning, which nothing else records, and never
787        // stated settle rules. Every other kind's brief is rebuilt from the
788        // question and today's state for each turn, so a brief stored by an
789        // older build (with owner-evidence rules since removed) never comes
790        // back; the stored copy remains only as a record.
791        let brief = match kind_of(&q) {
792            Some(kind @ (Kind::Land | Kind::Release | Kind::Triage | Kind::Generic)) => {
793                let state = (kind == Kind::Land)
794                    .then(|| crate::run::RunState::load(&q.run).ok())
795                    .flatten();
796                brief_for(&self.home, &q, kind, state.as_ref())
797            }
798            Some(Kind::Conduct) | None => dep.brief.clone(),
799        };
800        let current_brief = format!(
801            "{}\n\n{}\n\n{}",
802            brief,
803            crate::choice_effect::brief(&q),
804            crate::queue::dependency_facts(
805                &crate::queue::Queue::at(self.home.join("queue")),
806                &self.store
807            )
808        );
809        let body = prompt::deputy(&prompt::DeputyPrompt {
810            id: &q.id,
811            summary: &q.summary,
812            detail: &q.detail,
813            brief: &current_brief,
814            choices: &q.choices,
815            thread: read,
816            unread: unread.as_deref(),
817            resumed,
818            handover: false,
819            kind: kind_of(&q).unwrap_or(Kind::Conduct),
820            language: &cfg.graph.language,
821        });
822
823        // Recorded before the turn starts, so a daemon that dies mid-turn
824        // leaves a start counted and the seat on the record.
825        let next_start = next_start_path(&self.store, &q.id);
826        std::fs::create_dir_all(next_start.parent().expect("deputy artifacts parent"))?;
827        std::fs::write(
828            next_start,
829            now.as_second()
830                .saturating_add(RESTART_AFTER.as_secs() as i64)
831                .to_string(),
832        )?;
833        self.store.beat(&q.id, WaiterKind::Deputy);
834        let starts = dep.starts + 1;
835        let first = seat.clone();
836        self.store.update(&q.id, |r| {
837            if let Some(d) = r.deputy.as_mut() {
838                d.agent = spec.id.clone();
839                d.seat = Some(first);
840                d.starts = starts;
841            }
842            r.waiter = Some(Note {
843                kind: WaiterKind::Deputy,
844                since: now,
845            });
846            Ok(())
847        })?;
848        drop(release);
849        tracing::info!(
850            "question {}: deputy seat {} {} (start {starts}/{MAX_STARTS})",
851            q.short(),
852            seat.key,
853            if resumed { "resuming" } else { "starting" }
854        );
855
856        let left = (deadline(&q, cfg.graph.answer_timeout) - now.as_second()).max(0) as u64;
857        let artifacts = self.store.root().join(format!("{}.deputy", q.id));
858        let cache_dir = cfg.cache_dir();
859        // `magi ask` writes the question record and its lock, which a read-only
860        // sandbox refuses, so the seat cannot be read-only. It edits no
861        // repository files; the daemon, not the deputy, applies outcomes.
862        let allow_write = true;
863        // The question store is outside the repository, and `magi ask` writes it.
864        // A deputy also files the follow-up tasks the owner asks for with
865        // `magi task add`, which writes the queue; that is the one other place
866        // it writes.
867        let writable = [self.store.root().to_path_buf(), self.home.join("queue")];
868        macro_rules! invocation {
869            ($prompt:expr, $stem:expr, $timeout:expr) => {
870                Invocation {
871                    coordination: cfg.coordination.as_ref(),
872                    fetched_base: None,
873                    cwd: &cwd,
874                    prompt: $prompt,
875                    timeout: $timeout,
876                    allow_write,
877                    unsandboxed: false,
878                    sessions: cfg.graph.sessions,
879                    artifacts: &artifacts,
880                    stem: $stem,
881                    // The question's own run key (the task id) is what lets this
882                    // seat's `magi ask` pass the ownership check on a conductor
883                    // question.
884                    run: &q.run,
885                    node: NODE,
886                    cache_dir: cache_dir.as_deref(),
887                    attachments: &[],
888                    writable: &writable,
889                }
890            };
891        }
892
893        // A fresh seat first takes a short turn that ends: `agent::invoke` only
894        // learns the CLI's session id when it returns, and the real turn blocks
895        // in `magi ask` for hours, so a daemon stopped mid-wait would otherwise
896        // leave nothing to resume. This turn persists the seat before the wait.
897        let mut early = None;
898        if !resumed && cfg.graph.sessions {
899            let hbody = prompt::deputy(&prompt::DeputyPrompt {
900                id: &q.id,
901                summary: &q.summary,
902                detail: &q.detail,
903                brief: &current_brief,
904                choices: &q.choices,
905                thread: read,
906                unread: None,
907                resumed: false,
908                handover: true,
909                kind: kind_of(&q).unwrap_or(Kind::Conduct),
910                language: &cfg.graph.language,
911            });
912            let hstem = format!("handover-{starts}");
913            // Bounded by the question's own deadline, not only by its own cap: the
914            // lease it beats would otherwise keep an expired question alive.
915            let hlimit = HANDOVER_TIMEOUT.min(Duration::from_secs(left.max(1)));
916            let hinv = invocation!(&hbody, &hstem, hlimit);
917            let Some(done) = self.drive(&spec, &mut seat, &hinv, &q.id).await else {
918                self.park(&q.id);
919                return Ok(());
920            };
921            let kept = seat.clone();
922            self.store.update(&q.id, |r| {
923                if let Some(d) = r.deputy.as_mut() {
924                    d.seat = Some(kept);
925                }
926                Ok(())
927            })?;
928            if !matches!(&done, Ok(o) if o.usable()) {
929                early = Some(done);
930            }
931        }
932
933        // The handover may have been slow: look at the question again, and
934        // measure the long turn from what is left now.
935        let left = if early.is_none() && !resumed && cfg.graph.sessions {
936            let now = Timestamp::now();
937            let again = self.store.get(&q.id)?;
938            let left = (deadline(&again, cfg.graph.answer_timeout) - now.as_second()).max(0) as u64;
939            if !again.status.open() || (left == 0 && again.unread_from_owner().is_none()) {
940                self.park_quietly(&q.id);
941                return Ok(());
942            }
943            left
944        } else {
945            left
946        };
947        let out = match early {
948            Some(done) => done,
949            None => {
950                let stem = format!("turn-{starts}");
951                let timeout = Duration::from_secs(left.max(60) + SLACK_SECS);
952                let inv = invocation!(&body, &stem, timeout);
953                match self.drive(&spec, &mut seat, &inv, &q.id).await {
954                    Some(out) => out,
955                    None => {
956                        self.park(&q.id);
957                        return Ok(());
958                    }
959                }
960            }
961        };
962
963        let (text, why) = match out {
964            Ok(o) if o.usable() => (Some(o.text.trim().to_owned()), None),
965            Ok(o) if o.timed_out => (None, Some("its turn timed out".to_owned())),
966            Ok(o) if o.quota_exhausted() => (None, Some("the agent is out of quota".to_owned())),
967            Ok(o) => (
968                None,
969                Some(format!(
970                    "its turn failed ({})",
971                    notices::exit_status(o.exit_code)
972                )),
973            ),
974            Err(e) => (None, Some(format!("the agent could not be started: {e:#}"))),
975        };
976        let kept = seat.clone();
977        self.store.update(&q.id, |r| {
978            if let Some(d) = r.deputy.as_mut() {
979                d.seat = Some(kept);
980            }
981            r.waiter = None;
982            // Retire ownership under the same lock used by text admission.
983            self.store.drop_lease(&q.id);
984            // The owner's words were in the prompt. An agent that read them but
985            // answered in prose instead of `magi ask --thread` still spoke;
986            // keep it on the record so the owner reads it.
987            if let Some(text) = text
988                && unread.is_some()
989                && !text.is_empty()
990                && r.status.open()
991                && r.thread.len() == snapshot
992            {
993                r.delivered_turns = r.delivered_turns.max(snapshot);
994                let choices = r.choices.clone();
995                r.reply(text, choices)?;
996            }
997            Ok(())
998        })?;
999        if let Some(why) = why {
1000            tracing::warn!("question {}: the deputy ended: {why}", q.short());
1001            notices::raise_in(
1002                &self.home,
1003                Notice::warn(
1004                    &format!("deputy-turn:{}", q.id),
1005                    format!(
1006                        "Question {} \"{}\": its follow-up agent stopped ({why}).",
1007                        q.short(),
1008                        q.summary
1009                    ),
1010                )
1011                .link(if q.run_names_task() {
1012                    notices::Link::Task { id: q.run.clone() }
1013                } else {
1014                    notices::Link::Run { id: q.run.clone() }
1015                }),
1016            );
1017        }
1018        Ok(())
1019    }
1020}
1021
1022#[cfg(test)]
1023mod worker_tests {
1024    use super::*;
1025
1026    #[test]
1027    fn restart_cooldown_survives_a_new_runner() {
1028        let home = tempfile::tempdir().expect("home");
1029        let questions = Questions::at(home.path().join("questions"));
1030        let path = next_start_path(&questions, "question");
1031        std::fs::create_dir_all(path.parent().expect("parent")).expect("artifacts");
1032        std::fs::write(&path, "130").expect("next start");
1033        let reopened = Questions::at(home.path().join("questions"));
1034        assert!(!start_due(
1035            &reopened,
1036            "question",
1037            Timestamp::from_second(129).unwrap()
1038        ));
1039        assert!(start_due(
1040            &reopened,
1041            "question",
1042            Timestamp::from_second(130).unwrap()
1043        ));
1044    }
1045}