Skip to main content

magi/
triage.rs

1//! Held-task triage: walking `held` tasks so a hold left by an accident does
2//! not sit unread forever next to one a human placed on purpose.
3//!
4//! [`crate::queue::HoldSource`] already distinguishes "the daemon or
5//! conductor held this during its own recovery" ([`HoldSource::Machine`],
6//! documented as recoverable) from "an operator held this on purpose"
7//! ([`HoldSource::Manual`]). What was missing was anything that actually acts
8//! on that distinction: nothing walked the held list and asked whether a
9//! machine hold's cause was still true, and a record written before
10//! `hold_source` existed (schema < 3, `None`) was silently protected forever
11//! by [`Task::operator_held`]'s conservative default - never wrong, but also
12//! never looked at again by anything.
13//!
14//! [`run_once`] is that walk. For every `held` task it finds:
15//!
16//! - [`HoldSource::Machine`]: if [`machine_cause_resolved`] can tell the
17//!   cause is gone, the task goes straight back to `queued` - the same effect
18//!   as `magi task release`, just automatic. When it cannot tell, a
19//!   [`Question`] is filed once and the task stays held until answered.
20//! - `None` (a legacy record, or a hold nobody explained): always a question,
21//!   exactly once - the whole point being that "protected forever" must not
22//!   mean "never shown to anyone" either.
23//! - [`HoldSource::Manual`]: never touched automatically. Only once the hold
24//!   has sat untouched past [`MANUAL_STALE_AFTER`] does it earn a question of
25//!   its own, asking whether it is still wanted.
26//!
27//! Every question this module files carries [`NODE`] and uses the task id as
28//! [`Question::run`] - the same convention `crate::conduct`'s own questions
29//! use for a task rather than a run (see `conduct::apply_one`'s own comment
30//! on why the dedupe check there also filters on `node`, not `run` alone: an
31//! ordinary graph question's `run` is a real run id, and a coincidental
32//! equality with some task's id must not be read as "about this task").
33//! [`latest_triage_question`] follows the identical rule.
34//!
35//! # Why answers apply here rather than through `crate::queue::Task::block`
36//!
37//! `crate::conduct` blocks a task on its own question
38//! (`Task::block(vec![question_id], …)`), and `crate::daemon::resolve_blockers`
39//! unblocks it - unconditionally, back to `queued` - the moment that question
40//! is answered, whatever the answer actually said. That is correct for
41//! `conduct`: the *content* of the answer is meant for whoever reads
42//! `Task::answers` next, not for the resolver.
43//!
44//! A triage question's answer is different: "not yet" and "discard it" do two
45//! entirely different, non-resuming things, and only "resume it" may put the
46//! task back in line. Reusing the generic blocked/unblock path would resume
47//! every answer alike, so this module never calls [`Task::block`] and never
48//! leaves a triaged task anything but `held` while its question is open.
49//! [`interpret_answer`] reads [`Question::resolution`] itself and
50//! [`run_once`] acts on it directly: [`Task::release`] for an actual "resume
51//! it" choice, [`Queue::remove`] for "discard it" (捨ててよい really means
52//! "you may throw this away", not "leave it sitting held" - the English
53//! wording must say the same thing, not "leave it held"), and
54//! [`Task::hold_manual`] for anything else - which both keeps the task held
55//! and reclassifies it as a hold an operator has now actually seen, one
56//! `crate::conduct` and a later triage pass leave alone.
57//!
58//! The choice is read by its **position** in [`Question::choices`]
59//! ([`Wording::choices3`]/[`Wording::choices2`] always put "resume" first and
60//! "discard" third), never by comparing the answer text against [`Wording`]'s
61//! own strings picked from whatever config is in force *now* - the language a
62//! question was filed in and the language a later `run_once` call happens to
63//! read back are not guaranteed to be the same call's [`Config`], and a text
64//! comparison would silently misread a real "resume" answer as "keep held"
65//! the moment they disagree.
66//!
67//! # Idempotency
68//!
69//! [`run_once`] runs on every daemon idle tick (see `crate::daemon::poll`)
70//! and on every `magi task triage`, so an answered question must be applied to
71//! a task **at most once**, and a *fresh* question for the same task (once it
72//! is held again, or goes stale) must still be possible. The record is
73//! [`Task::triage_applied`], the question ids already applied. It cannot live
74//! in [`Task::hold_reason`]: [`Task::release`] clears that, so a "resume"
75//! answer left no trace, and a released task that failed back to `held` was
76//! released again by the same old answer with its attempts reset - forever.
77//! A task that comes back to `held` after an applied answer is therefore a
78//! new hold, handled per [`HoldSource`] (a fresh question for a machine hold).
79//! [`already_applied`] also still reads the `[triage:<short>]` marker
80//! [`keep_held_note`] appends to the hold reason, for records that pre-date
81//! the field.
82
83use std::path::{Path, PathBuf};
84use std::time::Duration;
85
86use jiff::Timestamp;
87
88use crate::ask::{Question, QuestionStatus, Questions};
89use crate::config::Config;
90use crate::disk;
91use crate::queue::{HoldSource, OperatorResume, Queue, Task, TaskStatus};
92
93/// Node recorded on every question this module files - `crate::conduct::NODE`
94/// for the same idea applied to a `crate::conduct` decision instead.
95pub const NODE: &str = "triage";
96
97/// Seat name on a filed question. Not a real agent seat - there is no model
98/// call anywhere in this module - but every [`Question`] needs one, and every
99/// other deterministic filer (`crate::land`'s merge approval) names itself
100/// the same way.
101const SEAT: &str = "triage";
102
103/// How long a [`HoldSource::Manual`] hold sits untouched before triage asks
104/// whether it is still wanted.
105///
106/// A judgement call, not a `magi.toml` setting - the same reasoning
107/// `ask::REPLY_QUIET_WINDOW` documents for itself: there is no operator
108/// preference for "how long is too long to ignore my own hold" that a
109/// per-repository config could be *right* about. Seven days is long enough
110/// that an ordinary multi-day hold (waiting on a dependency, waiting on the
111/// operator's own schedule) never gets nagged, and short enough that a hold
112/// nobody has looked at in a week surfaces again rather than aging into the
113/// kind of silent backlog this feature exists to prevent.
114const MANUAL_STALE_AFTER: Duration = Duration::from_secs(7 * 24 * 60 * 60);
115
116/// Localised strings for a filed question, the same idea as `crate::land`'s
117/// own `Words`/`words()` - only the languages magi can actually check are
118/// translated, and anything else falls back to English.
119struct Wording {
120    lang: &'static str,
121    resume: &'static str,
122    wait: &'static str,
123    discard: &'static str,
124    resume_now: &'static str,
125    keep_held: &'static str,
126}
127
128const EN: Wording = Wording {
129    lang: "en",
130    resume: "resume it",
131    wait: "not yet",
132    discard: "discard it",
133    resume_now: "resume it",
134    keep_held: "keep it held",
135};
136
137const JA: Wording = Wording {
138    lang: "ja",
139    resume: "再開してよい",
140    wait: "まだ待って",
141    discard: "捨ててよい",
142    resume_now: "再開する",
143    keep_held: "まだ止めておく",
144};
145
146/// Pick the wording. Codes and names both, the same acceptance
147/// `crate::land::words` gives `[graph] language`.
148fn wording(language: &str) -> &'static Wording {
149    let l = language.trim();
150    if l.eq_ignore_ascii_case("ja")
151        || l.eq_ignore_ascii_case("jp")
152        || l.eq_ignore_ascii_case("japanese")
153        || l.eq_ignore_ascii_case("日本語")
154    {
155        &JA
156    } else {
157        &EN
158    }
159}
160
161impl Wording {
162    fn choices3(&self) -> Vec<String> {
163        vec![
164            self.resume.to_owned(),
165            self.wait.to_owned(),
166            self.discard.to_owned(),
167        ]
168    }
169
170    fn choices2(&self) -> Vec<String> {
171        vec![self.resume_now.to_owned(), self.keep_held.to_owned()]
172    }
173
174    fn source_label(&self, source: Option<HoldSource>) -> &'static str {
175        match (self.lang, source) {
176            ("ja", Some(HoldSource::Machine)) => "machine(機械による自動保留)",
177            ("ja", Some(HoldSource::Manual)) => "manual(操作者による手動保留)",
178            ("ja", None) => "unknown(schema 3 未満の旧レコード、または理由未記録)",
179            (_, Some(HoldSource::Machine)) => "machine (automatic recovery hold)",
180            (_, Some(HoldSource::Manual)) => "manual (an operator held this)",
181            (_, None) => "unknown (pre-schema-3 record, or never recorded)",
182        }
183    }
184
185    /// The body under the summary: everything an operator needs to judge this
186    /// without opening a terminal - id, title, hold reason, hold source.
187    ///
188    /// Falls back to [`Task::last_error`] when [`Task::hold_reason`] is empty:
189    /// the most common `HoldSource::Machine` hold of all - `Task::fail` once
190    /// attempts run out - only ever sets `last_error`, never `hold_reason`, so
191    /// reading `hold_reason` alone would leave the question blank for exactly
192    /// the case requirement 4 exists for.
193    fn detail(&self, task: &Task, why: &str) -> String {
194        let none = if self.lang == "ja" {
195            "(記録なし)"
196        } else {
197            "(none recorded)"
198        };
199        let reason = task
200            .hold_reason
201            .as_deref()
202            .or(task.last_error.as_deref())
203            .unwrap_or(none);
204        format!(
205            "task: {} ({})\ntitle: {}\nhold source: {}\nhold reason: {reason}\n\n{why}",
206            task.id,
207            task.short(),
208            task.title,
209            self.source_label(task.hold_source),
210        )
211    }
212
213    fn summary_machine_unknown(&self, task: &Task) -> String {
214        if self.lang == "ja" {
215            format!("保留タスク {} の再開可否を判断してください", task.short())
216        } else {
217            format!("decide whether to resume held task {}", task.short())
218        }
219    }
220
221    fn why_machine(&self) -> &'static str {
222        if self.lang == "ja" {
223            "機械的な保留(machine hold)ですが、原因がすでに解消しているかを自動では判断できませんでした。"
224        } else {
225            "This is a machine hold, but whether its cause has resolved could not be \
226             checked automatically."
227        }
228    }
229
230    fn summary_conductor_override(&self, task: &Task) -> String {
231        if self.lang == "ja" {
232            format!(
233                "再開と回答済みのタスク {} を conductor が再び保留しました",
234                task.short()
235            )
236        } else {
237            format!(
238                "task {} was resumed at your word, but the conductor held it again",
239                task.short()
240            )
241        }
242    }
243
244    fn why_conductor_override(&self, o: &OperatorResume) -> String {
245        let reason = o.conductor_rehold.as_deref().unwrap_or_default();
246        if self.lang == "ja" {
247            format!(
248                "{} に再開と回答済みですが、conductor が再び hold しました。conductor の理由: \
249                 {reason}\n\n強制再キューを選ぶと、以後 conductor はこのタスクを hold できません。",
250                o.at
251            )
252        } else {
253            format!(
254                "You answered \"resume\" at {}, but the conductor held the task again. \
255                 Its reason: {reason}\n\nForcing a requeue stops the conductor from \
256                 holding this task again.",
257                o.at
258            )
259        }
260    }
261
262    /// Positions match [`AnswerAction`]: 0 resume, 1 keep held, 2 discard.
263    fn choices_conductor_override(&self) -> Vec<String> {
264        if self.lang == "ja" {
265            vec![
266                "強制再キュー(conductor は再 hold 不可)".to_owned(),
267                "手動 hold のまま".to_owned(),
268                "捨ててよい".to_owned(),
269            ]
270        } else {
271            vec![
272                "force requeue (conductor must not hold again)".to_owned(),
273                "keep held (manual)".to_owned(),
274                "discard".to_owned(),
275            ]
276        }
277    }
278
279    fn summary_legacy(&self, task: &Task) -> String {
280        if self.lang == "ja" {
281            format!(
282                "hold_source が不明な保留タスク {} を確認してください",
283                task.short()
284            )
285        } else {
286            format!(
287                "held task {} has no recorded hold source - please take a look",
288                task.short()
289            )
290        }
291    }
292
293    fn why_legacy(&self) -> &'static str {
294        if self.lang == "ja" {
295            "hold_source が記録されていません。schema 3 より前のレコードか、理由が記録されなかった \
296             holdです。人が意図して止めたのか、クラッシュや強制再起動で宙に浮いただけなのか、\
297             このデータからは区別できません。"
298        } else {
299            "No hold_source was recorded - either a pre-schema-3 record, or a hold whose \
300             reason was never written down. Whether this was a deliberate hold or the \
301             leftover of a crash cannot be told from the data alone."
302        }
303    }
304
305    fn summary_manual_stale(&self, task: &Task, days: i64) -> String {
306        if self.lang == "ja" {
307            format!(
308                "{days}日間 保留されたままの手動保留タスク {} を確認してください",
309                task.short()
310            )
311        } else {
312            format!(
313                "held task {} has been on a manual hold for {days} day(s)",
314                task.short()
315            )
316        }
317    }
318
319    fn why_manual(&self) -> &'static str {
320        if self.lang == "ja" {
321            "操作者が明示的に止めた保留ですが、長期間そのままになっています。まだ止めておくか、\
322             再開するか教えてください。"
323        } else {
324            "An operator held this on purpose, but it has sat untouched for a while. Say \
325             whether to keep holding it or resume it."
326        }
327    }
328}
329
330/// Which of the three situations this module recognises a held task is in.
331#[derive(Debug, Clone, Copy, PartialEq, Eq)]
332enum Bucket {
333    /// `HoldSource::Machine`, cause not verifiably resolved.
334    MachineUnknown,
335    /// `HoldSource::Machine`, held by the conductor after the operator
336    /// answered "resume" (see [`Task::resume_override`]).
337    ConductorOverride,
338    /// `hold_source` is `None`.
339    Legacy,
340    /// `HoldSource::Manual`, held past [`MANUAL_STALE_AFTER`].
341    ManualStale,
342}
343
344/// What one [`run_once`] pass did, task ids in each list.
345#[derive(Debug, Clone, Default)]
346pub struct Report {
347    /// A `HoldSource::Machine` hold whose cause was found resolved, put back
348    /// in line automatically.
349    pub resumed: Vec<String>,
350    /// A fresh question was filed this pass.
351    pub asked: Vec<String>,
352    /// An operator's answer to an earlier triage question was applied.
353    pub answered: Vec<String>,
354    /// A `blocked` task whose `blocked_by` named an id that no longer exists
355    /// was moved to a machine hold this pass - see
356    /// [`quarantine_orphaned_blocked`]. Distinct from `asked`: the question
357    /// about it, if any, is filed in the same pass and only counted there.
358    pub quarantined: Vec<String>,
359}
360
361impl Report {
362    /// Is there nothing to report? Callers use this to skip logging an empty
363    /// pass rather than repeating "triaged 0 held task(s)" on every idle tick.
364    pub fn is_empty(&self) -> bool {
365        self.resumed.is_empty()
366            && self.asked.is_empty()
367            && self.answered.is_empty()
368            && self.quarantined.is_empty()
369    }
370}
371
372/// The repository a task's config and disk check should read from. Mirrors
373/// `crate::conduct::repo_for`'s own fallback, duplicated rather than shared
374/// because that one is private to its module and the two are one `if` each.
375fn repo_for(task: &Task) -> PathBuf {
376    if task.repo.as_os_str().is_empty() {
377        PathBuf::from(".")
378    } else {
379        task.repo.clone()
380    }
381}
382
383/// Recognise a `HoldSource::Machine` hold caused by the free-space gate
384/// (`crate::disk::gate`'s message, or `daemon::disk_gate`'s "could not
385/// measure" fallback) from `hold_reason` text alone.
386///
387/// There is no field recording *why* a machine hold happened - `hold_machine`
388/// takes only a reason string - so this is the one signal available, and disk
389/// pressure is the one cause this module can safely re-measure without
390/// touching git, a run, or an agent CLI. Keep the two prefixes here in sync
391/// with `crate::disk::gate`'s formatted string and `daemon::disk_gate`'s own
392/// message if either changes; nothing else ties them together.
393fn is_disk_hold(task: &Task) -> bool {
394    task.hold_reason.as_deref().is_some_and(|r| {
395        r.starts_with("not enough free space to start a run:")
396            || r.starts_with("could not measure free space on ")
397    })
398}
399
400/// Has a `HoldSource::Machine` hold's cause resolved? `Some(true)` means yes -
401/// safe to requeue. `Some(false)` means the same cause was checked and is
402/// still in force. `None` means this hold's cause is not one this module
403/// knows how to re-check at all, and a human has to look.
404fn machine_cause_resolved(task: &Task, cfg: &Config) -> Option<bool> {
405    if !is_disk_hold(task) {
406        return None;
407    }
408    let min = cfg.disk.min_free_bytes;
409    if min == 0 {
410        // The operator turned the gate off since this hold was placed - its
411        // one possible cause is gone by construction, no measurement needed.
412        return Some(true);
413    }
414    let free = disk::free_bytes(&repo_for(task)).ok()?;
415    Some(disk::gate(free, min).is_none())
416}
417
418/// Is a `HoldSource::Manual` hold old enough to earn a "still wanted?"
419/// question? Same comparison `crate::clean::due` uses for a run's fold grace,
420/// against [`MANUAL_STALE_AFTER`] instead of a configured one.
421fn manual_is_stale(task: &Task, now: Timestamp) -> bool {
422    now.as_second() - task.updated_at.as_second() > MANUAL_STALE_AFTER.as_secs() as i64
423}
424
425/// The marker [`apply_answer`] writes into [`Task::hold_reason`] and
426/// [`already_applied`] reads back - see this module's own doc on why.
427fn marker_for(q: &Question) -> String {
428    format!("[triage:{}]", q.short())
429}
430
431/// Has `q`'s answer already been applied to `task`? See this module's doc.
432/// Checks [`Task::triage_applied`] first, which survives [`Task::release`].
433fn already_applied(task: &Task, q: &Question) -> bool {
434    if task.triage_applied(&q.id) {
435        return true;
436    }
437    // Records written before `Task::triage_applied` existed carry only the
438    // "keep held" marker in the hold reason.
439    let marker = marker_for(q);
440    task.hold_reason
441        .as_deref()
442        .is_some_and(|r| r.contains(marker.as_str()))
443}
444
445/// The most recent question this module filed for `task_id`, any status -
446/// open (still waiting), answered (may need applying), or abandoned (settled
447/// with nothing decided). Filters on both `node` and `run`, never `run`
448/// alone - see this module's doc on why a bare `run` match is not safe.
449fn latest_triage_question(questions: &Questions, task_id: &str) -> Option<Question> {
450    questions
451        .list()
452        .into_iter()
453        .filter(|q| q.node == NODE && q.run == task_id)
454        // `asked_at` first: ids carry only whole seconds plus a random
455        // suffix, so two questions filed in the same second order randomly.
456        .max_by(|a, b| a.asked_at.cmp(&b.asked_at).then_with(|| a.id.cmp(&b.id)))
457}
458
459/// What an answered triage question's choice means, independent of which
460/// language it was filed in.
461#[derive(Debug, Clone, Copy, PartialEq, Eq)]
462enum AnswerAction {
463    /// The first choice, always "resume it" / "再開してよい" / "再開する" -
464    /// see [`Wording::choices3`] and [`Wording::choices2`], whose first entry
465    /// is always the resume choice.
466    Resume,
467    /// The third choice, present only in [`Wording::choices3`]: "discard it"
468    /// / "捨ててよい".
469    Discard,
470    /// Anything else: the second choice ("not yet" / "keep it held"), or an
471    /// answer that does not match any offered choice at all (should not
472    /// happen for a multiple-choice question, but the safe default is to
473    /// keep holding rather than guess at "resume").
474    KeepHeld,
475}
476
477/// Read `q`'s answer as an [`AnswerAction`].
478///
479/// Matched by **position in `q.choices`**, never by comparing the answer text
480/// against [`Wording`]'s own strings: [`Wording`] is picked from the task's
481/// *current* repository config, which can differ from whatever language was
482/// in force when the question was filed (a `--config` override on one `magi
483/// task triage` call and not the next, or the repository's config edited in
484/// between). Comparing text would then silently misread an actual "resume"
485/// answer as "keep held" - the choices themselves are fixed at filing time,
486/// in [`file_question`], and never change afterwards, so their position is
487/// the one thing that stays true regardless of which language reads them
488/// back.
489fn interpret_answer(q: &Question) -> AnswerAction {
490    let resolution = q.resolution().unwrap_or_default();
491    match q.choices.iter().position(|c| *c == resolution) {
492        Some(0) => AnswerAction::Resume,
493        Some(2) => AnswerAction::Discard,
494        _ => AnswerAction::KeepHeld,
495    }
496}
497
498/// The note [`already_applied`] looks for, appended to (never replacing)
499/// whatever [`Task::hold_reason`] already said - the original cause is still
500/// worth reading in `magi task show` after the operator answers "not yet",
501/// and [`Task::hold_manual`] would otherwise overwrite it outright.
502fn keep_held_note(task: &Task, q: &Question, resolution: &str) -> String {
503    let marker = format!("{} operator: {resolution}", marker_for(q));
504    match task.hold_reason.as_deref() {
505        Some(existing) if !existing.is_empty() => format!("{existing}\n{marker}"),
506        _ => marker,
507    }
508}
509
510/// File a fresh triage question for `task` and return it. The caller is
511/// responsible for having already established there is no open one - see
512/// [`latest_triage_question`] - so this never checks again.
513fn file_question(
514    questions: &Questions,
515    task: &Task,
516    bucket: Bucket,
517    w: &Wording,
518    now: Timestamp,
519) -> Option<Question> {
520    let (summary, why, choices) = match bucket {
521        Bucket::MachineUnknown => (
522            w.summary_machine_unknown(task),
523            w.why_machine().to_owned(),
524            w.choices3(),
525        ),
526        Bucket::ConductorOverride => (
527            w.summary_conductor_override(task),
528            task.resume_override
529                .as_ref()
530                .map(|o| w.why_conductor_override(o))
531                .unwrap_or_default(),
532            w.choices_conductor_override(),
533        ),
534        Bucket::Legacy => (
535            w.summary_legacy(task),
536            w.why_legacy().to_owned(),
537            w.choices3(),
538        ),
539        Bucket::ManualStale => {
540            let days = (now.as_second() - task.updated_at.as_second()) / (24 * 60 * 60);
541            (
542                w.summary_manual_stale(task, days),
543                w.why_manual().to_owned(),
544                w.choices2(),
545            )
546        }
547    };
548    let mut q = Question::new(
549        task.id.clone(),
550        NODE.to_owned(),
551        SEAT.to_owned(),
552        summary,
553        w.detail(task, &why),
554        choices,
555    );
556    questions.put(&mut q).ok()?;
557    Some(q)
558}
559
560/// Move every `blocked` task whose `blocked_by` names a task or question id
561/// that no longer exists to a machine hold, before the per-`held` walk
562/// [`run_once`] does gets a look at it.
563///
564/// `crate::daemon::resolve_blockers` already catches the same situation on
565/// every idle poll, and [`Queue::remove`] already catches it the moment a
566/// dependency is deleted through `magi task rm` - both call the same
567/// [`crate::queue::missing_blockers`]/[`crate::queue::missing_blocker_hold_reason`]
568/// this does. This third copy exists because a dependency can also be deleted
569/// by hand (the file just removed from disk, not through either of those
570/// paths), and because a queue can carry a `blocked` task with a
571/// long-since-deleted dependency from *before* either catch above ever
572/// existed - and such a task is `blocked`, never `held`, so it is invisible
573/// to the rest of this module without this pass. Running it here, first, is
574/// also what makes `magi task triage` alone - with no daemon running at all -
575/// enough to fix one: the task lands `held` in this same call, and the
576/// ordinary loop below files its question in the very same pass.
577fn quarantine_orphaned_blocked(queue: &Queue, questions: &Questions) -> Vec<String> {
578    let mut quarantined = Vec::new();
579    for listed in queue.list() {
580        if listed.status != TaskStatus::Blocked || listed.blocked_by.is_empty() {
581            continue;
582        }
583        let Ok(_claim) = queue.claim(&listed.id) else {
584            continue;
585        };
586        let Ok(mut task) = queue.get(&listed.id) else {
587            continue;
588        };
589        if task.status != TaskStatus::Blocked {
590            continue;
591        }
592        let missing = crate::queue::missing_blockers(queue, questions, &task.blocked_by);
593        if missing.is_empty() {
594            continue;
595        }
596        task.hold_machine(Some(crate::queue::missing_blocker_hold_reason(
597            &task.blocked_by,
598            &missing,
599        )));
600        if queue.put(&mut task).is_ok() {
601            quarantined.push(task.id.clone());
602        }
603    }
604    quarantined
605}
606
607/// Run one deterministic triage pass over every `held` task in `queue`. No
608/// model call anywhere in this function - see this module's own doc for what
609/// each `HoldSource` gets instead.
610///
611/// `config_override` is threaded straight to [`Config::discover`], the same
612/// role `daemon::Opts::config` plays for `daemon::prepare` - an explicit
613/// `--config` from the caller, or `None` to let each task's own repository
614/// pick its layers.
615///
616/// Safe to call on every daemon idle tick and from `magi task triage` alike:
617/// a task already answered and applied is left alone (see
618/// [`already_applied`]), and a task with an open question is left alone too,
619/// so repeated calls with nothing new to say do nothing.
620///
621/// Also runs [`quarantine_orphaned_blocked`] first, so a `blocked` task whose
622/// dependency no longer exists is caught and turned into a fresh `held`
623/// question in this same pass, not left for a later call to notice.
624pub fn run_once(
625    queue: &Queue,
626    questions: &Questions,
627    config_override: Option<&Path>,
628    now: Timestamp,
629) -> Report {
630    let mut report = Report {
631        quarantined: quarantine_orphaned_blocked(queue, questions),
632        ..Report::default()
633    };
634    for listed in queue.list() {
635        if listed.status != TaskStatus::Held {
636            continue;
637        }
638        let Ok(_claim) = queue.claim(&listed.id) else {
639            continue;
640        };
641        let Ok(mut task) = queue.get(&listed.id) else {
642            continue;
643        };
644        // Re-read under the claim: a release or a re-hold landed by a human
645        // between the listing above and the claim just taken must not be
646        // clobbered by a decision based on the stale copy.
647        if task.status != TaskStatus::Held {
648            continue;
649        }
650
651        let cfg = Config::discover(&repo_for(&task), config_override)
652            .ok()
653            .map(|(c, _)| c);
654        let w = wording(cfg.as_ref().map_or("en", |c| c.graph.language.as_str()));
655
656        if let Some(q) = latest_triage_question(questions, &task.id) {
657            if q.status.open() {
658                // Already asked, still waiting - nothing to do this pass.
659                continue;
660            }
661            if q.status == QuestionStatus::Answered && !already_applied(&task, &q) {
662                match interpret_answer(&q) {
663                    AnswerAction::Resume => {
664                        // A second "resume", to the question about the
665                        // conductor's re-hold, forces it: the conductor may
666                        // not hold this task again. Any other resume records
667                        // the answer so a re-hold can be recognised.
668                        let contradicted = task
669                            .resume_override
670                            .as_ref()
671                            .is_some_and(|o| o.conductor_rehold.is_some());
672                        let record = match task.resume_override.take() {
673                            Some(mut o) if contradicted => {
674                                o.forced = true;
675                                o
676                            }
677                            _ => OperatorResume {
678                                question_id: q.id.clone(),
679                                at: now,
680                                conductor_rehold: None,
681                                forced: false,
682                            },
683                        };
684                        task.release();
685                        task.resume_override = Some(record);
686                        task.mark_triage_applied(&q.id);
687                        if queue.put(&mut task).is_ok() {
688                            report.answered.push(task.id.clone());
689                        }
690                    }
691                    AnswerAction::Discard => {
692                        if queue.remove(&task.id, false, questions).is_ok() {
693                            report.answered.push(task.id.clone());
694                        }
695                    }
696                    AnswerAction::KeepHeld => {
697                        let resolution = q.resolution().unwrap_or_default();
698                        let note = keep_held_note(&task, &q, &resolution);
699                        task.hold_manual(Some(note));
700                        task.mark_triage_applied(&q.id);
701                        if queue.put(&mut task).is_ok() {
702                            report.answered.push(task.id.clone());
703                        }
704                    }
705                }
706                continue;
707            }
708            // Abandoned, or an already-applied answer: fall through to the
709            // ordinary per-source handling below, which is how a stale
710            // `HoldSource::Manual` re-ask - or a fresh machine/legacy
711            // question, once a prior one settled the task back into a hold -
712            // gets filed.
713        }
714
715        match task.hold_source {
716            Some(HoldSource::Machine) => {
717                let overridden = task
718                    .resume_override
719                    .as_ref()
720                    .is_some_and(|o| o.conductor_rehold.is_some() && !o.forced);
721                if overridden {
722                    if file_question(questions, &task, Bucket::ConductorOverride, w, now).is_some()
723                    {
724                        report.asked.push(task.id.clone());
725                    }
726                } else if cfg.as_ref().and_then(|c| machine_cause_resolved(&task, c)) == Some(true)
727                {
728                    task.release();
729                    if queue.put(&mut task).is_ok() {
730                        report.resumed.push(task.id.clone());
731                    }
732                } else if file_question(questions, &task, Bucket::MachineUnknown, w, now).is_some()
733                {
734                    report.asked.push(task.id.clone());
735                }
736            }
737            None => {
738                if file_question(questions, &task, Bucket::Legacy, w, now).is_some() {
739                    report.asked.push(task.id.clone());
740                }
741            }
742            Some(HoldSource::Manual) => {
743                if manual_is_stale(&task, now)
744                    && file_question(questions, &task, Bucket::ManualStale, w, now).is_some()
745                {
746                    report.asked.push(task.id.clone());
747                }
748            }
749        }
750    }
751    report
752}
753
754/// The open triage question about `task_id`, if any - what `magi task show`
755/// prints so a held task's card names the question waiting on it, not only
756/// its hold reason. `None` once it is answered or abandoned: nothing is
757/// waiting on it anymore.
758pub fn open_question_for(questions: &Questions, task_id: &str) -> Option<Question> {
759    latest_triage_question(questions, task_id).filter(|q| q.status.open())
760}
761
762/// Does this module still have unfinished business with `task`?
763///
764/// True while its latest triage question is still open (waiting on an
765/// answer), and true for a beat longer than [`open_question_for`] alone
766/// would say: once answered, the question sits [`QuestionStatus::Answered`]
767/// but unread until the next [`run_once`] pass actually applies it (see
768/// [`already_applied`]), and [`run_once`] only ever runs on a fully idle
769/// daemon tick - far less often than `crate::conduct` polls. A caller that
770/// only checked "is a question open" would walk straight through that gap
771/// the moment the operator answers, moving the task out of `held` before
772/// [`run_once`] gets a turn - orphaning the very answer it was about to
773/// apply, the same failure mode this function exists to keep `crate::conduct`
774/// out of. `crate::conduct::apply_one` is exactly that caller.
775pub fn pending_for(questions: &Questions, task: &Task) -> bool {
776    match latest_triage_question(questions, &task.id) {
777        Some(q) if q.status.open() => true,
778        Some(q) if q.status == QuestionStatus::Answered => !already_applied(task, &q),
779        _ => false,
780    }
781}
782
783/// Every task id with an open triage question right now - what `magi task
784/// list` uses to mark a held task that is already waiting on an operator
785/// decision, rather than have it read identically to one nobody has looked
786/// at yet.
787pub fn open_task_ids(questions: &Questions) -> std::collections::BTreeSet<String> {
788    questions
789        .list()
790        .into_iter()
791        .filter(|q| q.node == NODE && q.status.open())
792        .map(|q| q.run)
793        .collect()
794}
795
796#[cfg(test)]
797mod tests {
798    use super::*;
799    use crate::ask::Answer;
800    use crate::queue::Source;
801    use jiff::SignedDuration;
802
803    fn store() -> (tempfile::TempDir, Queue, Questions) {
804        let dir = tempfile::tempdir().unwrap();
805        let q = Queue::at(dir.path().join("queue"));
806        let s = Questions::at(dir.path().join("questions"));
807        (dir, q, s)
808    }
809
810    fn task(title: &str, repo: PathBuf) -> Task {
811        Task::new(title.to_owned(), format!("do {title}"), repo, Source::Human)
812    }
813
814    /// `[disk] min_free_bytes = 0` written next to a fictional repo, so
815    /// `machine_cause_resolved` never has to ask the real disk anything - the
816    /// same fixture pattern `daemon`'s own idle-loop tests use.
817    fn gate_disabled_config(dir: &std::path::Path) -> PathBuf {
818        let config = dir.join("magi.toml");
819        std::fs::write(&config, "[disk]\nmin_free_bytes = 0\n").unwrap();
820        config
821    }
822
823    #[test]
824    fn a_resolved_machine_hold_is_requeued_automatically() {
825        let (dir, q, questions) = store();
826        let config = gate_disabled_config(dir.path());
827        let mut t = task("disk pressure", dir.path().join("repo"));
828        t.hold_machine(Some(
829            "not enough free space to start a run: 10 bytes free, 100 required by \
830             `[disk] min_free_bytes`"
831                .to_owned(),
832        ));
833        q.put(&mut t).unwrap();
834
835        let report = run_once(&q, &questions, Some(&config), Timestamp::now());
836        assert_eq!(report.resumed, [t.id.clone()]);
837        assert!(report.asked.is_empty());
838
839        let back = q.get(&t.id).unwrap();
840        assert_eq!(back.status, TaskStatus::Queued);
841        assert!(back.hold_source.is_none());
842        assert!(questions.list().is_empty(), "nothing needed asking");
843    }
844
845    #[test]
846    fn a_machine_hold_with_no_recognised_cause_gets_one_question_not_two() {
847        let (dir, q, questions) = store();
848        let mut t = task("gate went red", dir.path().join("repo"));
849        t.hold_machine(Some("gate red".to_owned()));
850        q.put(&mut t).unwrap();
851
852        let first = run_once(&q, &questions, None, Timestamp::now());
853        assert_eq!(first.asked, [t.id.clone()]);
854        assert!(first.resumed.is_empty());
855
856        let open: Vec<_> = questions
857            .list()
858            .into_iter()
859            .filter(|q| q.status.open())
860            .collect();
861        assert_eq!(open.len(), 1);
862        assert_eq!(open[0].run, t.id);
863        assert_eq!(open[0].node, NODE);
864        assert_eq!(open[0].choices.len(), 3);
865
866        // A second pass with nothing new must not file a second question.
867        let second = run_once(&q, &questions, None, Timestamp::now());
868        assert!(second.asked.is_empty());
869        assert_eq!(
870            questions
871                .list()
872                .into_iter()
873                .filter(|q| q.status.open())
874                .count(),
875            1
876        );
877    }
878
879    #[test]
880    fn a_legacy_hold_with_no_recorded_source_gets_exactly_one_question() {
881        let (dir, q, questions) = store();
882        let mut t = task("schema 1 record", dir.path().join("repo"));
883        t.status = TaskStatus::Held;
884        assert!(t.hold_source.is_none(), "the case this test is about");
885        q.put(&mut t).unwrap();
886
887        let first = run_once(&q, &questions, None, Timestamp::now());
888        assert_eq!(first.asked, [t.id.clone()]);
889
890        let second = run_once(&q, &questions, None, Timestamp::now());
891        assert!(
892            second.asked.is_empty(),
893            "the same legacy hold must not be asked about twice"
894        );
895        assert_eq!(
896            questions
897                .list()
898                .into_iter()
899                .filter(|q| q.status.open())
900                .count(),
901            1
902        );
903    }
904
905    #[test]
906    fn a_manual_hold_is_never_auto_resumed() {
907        let (dir, q, questions) = store();
908        let mut t = task("operator stopped this", dir.path().join("repo"));
909        t.hold_manual(Some("waiting on a decision".to_owned()));
910        q.put(&mut t).unwrap();
911
912        let report = run_once(&q, &questions, None, Timestamp::now());
913        assert!(report.resumed.is_empty());
914        // Fresh, not stale yet - no question either.
915        assert!(report.asked.is_empty());
916
917        let back = q.get(&t.id).unwrap();
918        assert_eq!(back.status, TaskStatus::Held);
919        assert_eq!(back.hold_source, Some(HoldSource::Manual));
920        assert!(questions.list().is_empty());
921    }
922
923    #[test]
924    fn a_blocked_task_on_a_deleted_dependency_is_held_and_asked_about_in_one_pass() {
925        // The five real tasks this whole change exists for are `blocked`, not
926        // `held`, and no daemon has to be running for `magi task triage` alone
927        // to reach them - `run_once` must both quarantine and ask in the same
928        // call.
929        let (dir, q, questions) = store();
930        let mut still_going = task("still valid", dir.path().join("repo"));
931        q.put(&mut still_going).unwrap();
932
933        let mut t = task("orphaned", dir.path().join("repo"));
934        t.block(
935            vec!["20260101-000000-gone".to_owned(), still_going.id.clone()],
936            Some("waits on both".to_owned()),
937        );
938        q.put(&mut t).unwrap();
939
940        let report = run_once(&q, &questions, None, Timestamp::now());
941        assert_eq!(report.quarantined, [t.id.clone()]);
942        assert_eq!(
943            report.asked,
944            [t.id.clone()],
945            "the fresh machine hold must earn a question in the same pass"
946        );
947
948        let after = q.get(&t.id).unwrap();
949        assert_eq!(after.status, TaskStatus::Held);
950        assert_eq!(after.hold_source, Some(HoldSource::Machine));
951        assert!(after.blocked_by.is_empty());
952
953        // The reason is not disk-pressure wording, so this must not be read
954        // as a disk hold and silently auto-resumed.
955        assert!(!is_disk_hold(&after));
956
957        let open: Vec<_> = questions
958            .list()
959            .into_iter()
960            .filter(|q| q.status.open())
961            .collect();
962        assert_eq!(open.len(), 1);
963        assert_eq!(open[0].run, t.id);
964
965        // A second pass with nothing new files no second question.
966        let second = run_once(&q, &questions, None, Timestamp::now());
967        assert!(second.quarantined.is_empty());
968        assert!(second.asked.is_empty());
969    }
970
971    #[test]
972    fn a_stale_manual_hold_earns_a_two_choice_question() {
973        let (dir, q, questions) = store();
974        let mut t = task("been sitting a while", dir.path().join("repo"));
975        t.hold_manual(Some("waiting on a decision".to_owned()));
976        q.put(&mut t).unwrap();
977        // Back-date the hold past MANUAL_STALE_AFTER without waiting a week.
978        let mut back = q.get(&t.id).unwrap();
979        back.updated_at = Timestamp::now() - SignedDuration::new(8 * 24 * 60 * 60, 0);
980        std::fs::write(
981            q.path_of(&back.id),
982            serde_json::to_string_pretty(&back).unwrap(),
983        )
984        .unwrap();
985
986        let report = run_once(&q, &questions, None, Timestamp::now());
987        assert_eq!(report.asked, [t.id.clone()]);
988        let open: Vec<_> = questions
989            .list()
990            .into_iter()
991            .filter(|q| q.status.open())
992            .collect();
993        assert_eq!(open.len(), 1);
994        assert_eq!(open[0].choices.len(), 2);
995    }
996
997    #[test]
998    fn answering_resume_releases_the_task() {
999        let (dir, q, questions) = store();
1000        let mut t = task("gate went red", dir.path().join("repo"));
1001        t.hold_machine(Some("gate red".to_owned()));
1002        q.put(&mut t).unwrap();
1003        run_once(&q, &questions, None, Timestamp::now());
1004
1005        let mut asked = questions
1006            .list()
1007            .into_iter()
1008            .find(|q| q.run == t.id)
1009            .unwrap();
1010        asked.answer(Answer::Choice(EN.resume.to_owned())).unwrap();
1011        questions.put(&mut asked).unwrap();
1012
1013        let report = run_once(&q, &questions, None, Timestamp::now());
1014        assert_eq!(report.answered, [t.id.clone()]);
1015        let back = q.get(&t.id).unwrap();
1016        assert_eq!(back.status, TaskStatus::Queued);
1017        assert!(back.hold_source.is_none());
1018    }
1019
1020    /// A task released by a "resume" answer, run, and failed back to `held`.
1021    fn resumed_then_failed(q: &Queue, questions: &Questions, dir: &std::path::Path) -> Task {
1022        let mut t = task("gate went red", dir.join("repo"));
1023        t.hold_machine(Some("gate red".to_owned()));
1024        q.put(&mut t).unwrap();
1025        run_once(q, questions, None, Timestamp::now());
1026        let mut asked = questions
1027            .list()
1028            .into_iter()
1029            .find(|q| q.run == t.id)
1030            .unwrap();
1031        asked.answer(Answer::Choice(EN.resume.to_owned())).unwrap();
1032        questions.put(&mut asked).unwrap();
1033
1034        let report = run_once(q, questions, None, Timestamp::now());
1035        assert_eq!(report.answered, [t.id.clone()]);
1036        let mut back = q.get(&t.id).unwrap();
1037        assert_eq!(back.status, TaskStatus::Queued);
1038        assert_eq!(back.attempts, 0);
1039
1040        back.start("run-1".to_owned());
1041        back.fail("rebase conflict", 1);
1042        assert_eq!(back.status, TaskStatus::Held);
1043        q.put(&mut back).unwrap();
1044        back
1045    }
1046
1047    #[test]
1048    fn a_resume_answer_is_applied_once_and_a_new_machine_hold_is_asked_about() {
1049        let (dir, q, questions) = store();
1050        let t = resumed_then_failed(&q, &questions, dir.path());
1051
1052        let report = run_once(&q, &questions, None, Timestamp::now());
1053        assert!(report.answered.is_empty(), "the old answer must not replay");
1054        assert_eq!(report.asked, std::slice::from_ref(&t.id));
1055        let back = q.get(&t.id).unwrap();
1056        assert_eq!(back.status, TaskStatus::Held);
1057        assert_eq!(
1058            questions
1059                .list()
1060                .into_iter()
1061                .filter(|q| q.status.open())
1062                .count(),
1063            1
1064        );
1065    }
1066
1067    #[test]
1068    fn a_manual_hold_placed_after_a_resume_is_not_undone_by_the_old_answer() {
1069        let (dir, q, questions) = store();
1070        let mut t = resumed_then_failed(&q, &questions, dir.path());
1071        t.hold_manual(Some("operator stopped this".to_owned()));
1072        q.put(&mut t).unwrap();
1073        let before = questions.list().len();
1074
1075        let report = run_once(&q, &questions, None, Timestamp::now());
1076        assert!(report.answered.is_empty());
1077        assert!(report.asked.is_empty());
1078        let back = q.get(&t.id).unwrap();
1079        assert_eq!(back.status, TaskStatus::Held);
1080        assert_eq!(back.hold_source, Some(HoldSource::Manual));
1081        assert_eq!(questions.list().len(), before);
1082    }
1083
1084    #[test]
1085    fn answering_not_yet_keeps_it_held_as_a_manual_hold_and_does_not_reapply() {
1086        let (dir, q, questions) = store();
1087        let mut t = task("gate went red", dir.path().join("repo"));
1088        t.hold_machine(Some("gate red".to_owned()));
1089        q.put(&mut t).unwrap();
1090        run_once(&q, &questions, None, Timestamp::now());
1091
1092        let mut asked = questions
1093            .list()
1094            .into_iter()
1095            .find(|q| q.run == t.id)
1096            .unwrap();
1097        asked.answer(Answer::Choice(EN.wait.to_owned())).unwrap();
1098        questions.put(&mut asked).unwrap();
1099
1100        let report = run_once(&q, &questions, None, Timestamp::now());
1101        assert_eq!(report.answered, [t.id.clone()]);
1102        let back = q.get(&t.id).unwrap();
1103        assert_eq!(back.status, TaskStatus::Held);
1104        assert_eq!(back.hold_source, Some(HoldSource::Manual));
1105        assert!(
1106            back.hold_reason
1107                .as_deref()
1108                .is_some_and(|r| r.contains("gate red")),
1109            "the original cause must survive a \"not yet\" answer, not just the \
1110             triage marker: {:?}",
1111            back.hold_reason
1112        );
1113
1114        // A third pass, same instant: the manual hold is fresh (just
1115        // touched), so nothing more happens - in particular the already
1116        // answered question is not re-applied.
1117        let third = run_once(&q, &questions, None, Timestamp::now());
1118        assert!(third.answered.is_empty());
1119        assert!(third.asked.is_empty());
1120    }
1121
1122    #[test]
1123    fn answering_discard_removes_the_task_entirely() {
1124        let (dir, q, questions) = store();
1125        let mut t = task("gate went red", dir.path().join("repo"));
1126        t.hold_machine(Some("gate red".to_owned()));
1127        q.put(&mut t).unwrap();
1128        run_once(&q, &questions, None, Timestamp::now());
1129
1130        let mut asked = questions
1131            .list()
1132            .into_iter()
1133            .find(|q| q.run == t.id)
1134            .unwrap();
1135        asked.answer(Answer::Choice(EN.discard.to_owned())).unwrap();
1136        questions.put(&mut asked).unwrap();
1137
1138        let report = run_once(&q, &questions, None, Timestamp::now());
1139        assert_eq!(report.answered, [t.id.clone()]);
1140        assert!(
1141            q.get(&t.id).is_err(),
1142            "\"discard it\" (捨ててよい) must actually discard the task, not \
1143             just leave it sitting held forever"
1144        );
1145    }
1146
1147    #[test]
1148    fn an_answer_is_read_by_its_position_in_choices_not_by_the_callers_current_language() {
1149        // Filed while the repo's config reads Japanese...
1150        let (dir, q, questions) = store();
1151        let ja_config = dir.path().join("ja.toml");
1152        std::fs::write(&ja_config, "[graph]\nlanguage = \"ja\"\n").unwrap();
1153        let mut t = task("gate went red", dir.path().join("repo"));
1154        t.hold_machine(Some("gate red".to_owned()));
1155        q.put(&mut t).unwrap();
1156        run_once(&q, &questions, Some(&ja_config), Timestamp::now());
1157
1158        let mut asked = questions
1159            .list()
1160            .into_iter()
1161            .find(|q| q.run == t.id)
1162            .unwrap();
1163        assert_eq!(asked.choices[0], JA.resume, "filed in Japanese");
1164        asked.answer(Answer::Choice(JA.resume.to_owned())).unwrap();
1165        questions.put(&mut asked).unwrap();
1166
1167        // ...but applied against an English config (a later `--config`, or an
1168        // edited repository config). The Japanese "再開してよい" answer must
1169        // still be read as a resume, not silently misread as "keep held"
1170        // because it fails a text comparison against the English wording.
1171        let en_config = dir.path().join("en.toml");
1172        std::fs::write(&en_config, "[graph]\nlanguage = \"en\"\n").unwrap();
1173        let report = run_once(&q, &questions, Some(&en_config), Timestamp::now());
1174        assert_eq!(report.answered, [t.id.clone()]);
1175        let back = q.get(&t.id).unwrap();
1176        assert_eq!(
1177            back.status,
1178            TaskStatus::Queued,
1179            "a resume answer must resume the task regardless of which \
1180             language it is read back in"
1181        );
1182    }
1183
1184    #[test]
1185    fn a_question_falls_back_to_last_error_when_hold_reason_was_never_set() {
1186        // `Task::fail` - the ordinary "out of attempts" machine hold - only
1187        // ever sets `last_error`, never `hold_reason`. The question detail
1188        // must still name a cause rather than reading "(none recorded)".
1189        let (dir, q, questions) = store();
1190        let mut t = task("kept failing the gate", dir.path().join("repo"));
1191        t.start("run-1".to_owned());
1192        t.fail("gate red three times running", 1);
1193        assert_eq!(t.status, TaskStatus::Held);
1194        assert!(t.hold_reason.is_none(), "the case this test is about");
1195        q.put(&mut t).unwrap();
1196
1197        run_once(&q, &questions, None, Timestamp::now());
1198        let asked = questions
1199            .list()
1200            .into_iter()
1201            .find(|q| q.run == t.id)
1202            .unwrap();
1203        assert!(
1204            asked.detail.contains("gate red three times running"),
1205            "the question must surface `last_error` when there is no \
1206             `hold_reason` to show instead: {}",
1207            asked.detail
1208        );
1209    }
1210
1211    #[test]
1212    fn open_question_for_and_open_task_ids_reflect_only_what_is_still_waiting() {
1213        let (dir, q, questions) = store();
1214        let mut t = task("gate went red", dir.path().join("repo"));
1215        t.hold_machine(Some("gate red".to_owned()));
1216        q.put(&mut t).unwrap();
1217
1218        assert!(open_question_for(&questions, &t.id).is_none());
1219        assert!(!open_task_ids(&questions).contains(&t.id));
1220
1221        run_once(&q, &questions, None, Timestamp::now());
1222        assert!(open_question_for(&questions, &t.id).is_some());
1223        assert!(open_task_ids(&questions).contains(&t.id));
1224
1225        let mut asked = questions
1226            .list()
1227            .into_iter()
1228            .find(|q| q.run == t.id)
1229            .unwrap();
1230        asked.answer(Answer::Choice(EN.resume.to_owned())).unwrap();
1231        questions.put(&mut asked).unwrap();
1232
1233        assert!(
1234            open_question_for(&questions, &t.id).is_none(),
1235            "an answered question is no longer open"
1236        );
1237        assert!(!open_task_ids(&questions).contains(&t.id));
1238    }
1239
1240    #[test]
1241    fn pending_for_stays_true_between_an_answer_and_the_next_run_once_pass() {
1242        // `open_question_for` alone goes `None` the instant the operator
1243        // answers, well before `run_once` - idle-tick only - gets a turn to
1244        // actually apply that answer (see `already_applied`). `pending_for`
1245        // exists so a caller polling far more often than `run_once` does -
1246        // `crate::conduct::apply_one` - does not walk through that gap.
1247        let (dir, q, questions) = store();
1248        let mut t = task("gate went red", dir.path().join("repo"));
1249        t.hold_machine(Some("gate red".to_owned()));
1250        q.put(&mut t).unwrap();
1251
1252        assert!(!pending_for(&questions, &t));
1253
1254        run_once(&q, &questions, None, Timestamp::now());
1255        let held = q.get(&t.id).unwrap();
1256        assert!(pending_for(&questions, &held), "still waiting on an answer");
1257
1258        let mut asked = questions
1259            .list()
1260            .into_iter()
1261            .find(|q| q.run == t.id)
1262            .unwrap();
1263        asked.answer(Answer::Choice(EN.wait.to_owned())).unwrap();
1264        questions.put(&mut asked).unwrap();
1265        assert!(!asked.status.open());
1266
1267        // The race window: answered, but `run_once` has not run again yet.
1268        let still_held = q.get(&t.id).unwrap();
1269        assert!(
1270            pending_for(&questions, &still_held),
1271            "answered but not yet applied is still pending"
1272        );
1273
1274        run_once(&q, &questions, None, Timestamp::now());
1275        let after = q.get(&t.id).unwrap();
1276        assert!(
1277            !pending_for(&questions, &after),
1278            "the answer is applied now, nothing left pending"
1279        );
1280    }
1281}