Skip to main content

task_runs/
driver.rs

1//! @arch:layer(kg_store)
2//! @arch:role(substrate)
3//! @arch:see(.yah/docs/working/yah-task-runs.md)
4//!
5//! PTY subprocess driver — spawn commands, capture output as append-only
6//! chunks, handle SIGTERM/SIGKILL with a grace period, and mark stale
7//! `Running` runs as `Lost` when the daemon restarts.
8//!
9//! ## Tier 2 side-channel (yah-log shims)
10//!
11//! When `SpawnOpts::log_fd_enabled` is true (the default), the driver creates
12//! a named pipe (FIFO) and exports two env vars into the child:
13//!
14//! - `YAH_TASK_RUN`  — the `TaskRunId` as a hyphenated UUID string.
15//! - `YAH_LOG_PIPE`  — absolute path to the FIFO.
16//!
17//! The child opens `YAH_LOG_PIPE` for writing and emits JSON-lines. The
18//! driver reads those lines in a background thread and stores them as
19//! [`EventSource::Shim`] events.
20//!
21//! **Why FIFO instead of a raw fd?** `portable-pty` calls `close_random_fds()`
22//! in its `pre_exec` hook, closing every fd ≥ 3 before exec. A raw-pipe write
23//! fd is always ≥ 3 and would be closed before the child could use it. Opening
24//! a FIFO by path requires no fd inheritance.
25//!
26//! Wire format — one JSON object per line:
27//! ```json
28//! {"level":"info","target":"myapp::module","msg":"text","fields":{"key":"val"}}
29//! ```
30//! Optional shim-identity keys: `"_lib"` (string), `"_lib_ver"` (string).
31//! Unknown keys in `fields` pass through as freeform JSON.
32//!
33//! The driver holds the write end of the FIFO open until the run lifecycle
34//! task completes, which triggers EOF for the receiver thread. The FIFO file
35//! is deleted after the receiver thread drains the last line.
36//!
37//! On non-Unix platforms `YAH_TASK_RUN` and `YAH_LOG_PIPE` are not exported.
38//! Shim libraries must treat absent `YAH_TASK_RUN` as "not inside a TaskRun".
39//!
40//! @yah:ticket(R617-F6, "Reattach-by-run_id replaces Lost-on-disappear for origin=terminal shells")
41//! @yah:status(review)
42//! @yah:assignee(agent:bundle-anthropic-ashguard)
43//! @yah:at(2026-07-24T01:26:41Z)
44//! @yah:phase(P3)
45//! @yah:parent(R617)
46//! @arch:see(.yah/docs/working/W280-durable-terminal-sessions.md)
47//! @yah:depends_on(R617-F13)
48//! @yah:handoff("DELIVERED. Verified: `cd oss/qed && cargo test -p task-runs --lib` 243/243 (was 237 — 6 new); `cargo test -p kg-daemon --lib shell_vt` 9/9; `cargo test -p yah --lib r617` 9/9; `cargo test -p desktop --lib` 357 pass / 2 fail, both pre-existing and in files this ticket does not touch (agent.rs rules-view expects 12 rows and a peer's approval-rule change makes 19; agent_process reader-finished is a known timing flake).")
49//! @yah:handoff("THE TICKET'S OWN FRAMING WAS WRONG ABOUT THE MECHANISM, and the correction is the design. @yah:next said to 're-adopt' a live shell by 'control channel rebuilt, reader thread restarted against the surviving PTY'. That is not possible and never was: you cannot re-open another process's PTY master fd. The real defect is narrower and worse — a driver was tombstoning runs IT DID NOT OWN. `.yah/db/task-runs.turso` has several writers (desktop, the R617-F13 shell host, one CampService per MCP sidecar), and `TaskDriver::new` assumed any leftover `Running` row must be its own predecessor's corpse. So every attach marked some other LIVE process's shell `Lost`, and that shell kept producing output under a status saying it was dead. The fix is therefore 'do not tombstone what you do not own', not 'reattach'. Actual PTY reattach is unnecessary once F13 puts the PTY in a process that outlives the desktop.")
50//! @yah:handoff("HOW OWNERSHIP IS KNOWN: new `TaskRunMeta::host_pid` — the pid of the process whose driver spawned the run, NOT the child's. Stamped by `spawn_run` at INSERT, before the child exists, so a crash between insert and spawn still leaves the row attributable. Store column added by the same idempotent `ALTER TABLE ... ADD COLUMN` pattern `origin` used, and `row_to_meta` reads index 15 with `.ok().flatten()` so a DB with no such column reads `None` rather than erroring.")
51//! @yah:handoff("THE SEAM IS ORIGIN-AGNOSTIC, per this ticket's gotcha. New `task_runs::StaleRunPolicy` in oss/qed/crates/task-runs/src/driver.rs: `LostOnDisappear` (the default — `TaskDriver::new` and `with_channels` behave exactly as before, so no existing embedder changed) and `AdoptLiveHosts { origins: Vec<String> }`, which spares a leftover run only when its `host_pid` names a process that still exists. The crate decides on OWNERSHIP and takes the origin list as data — it never learns what 'terminal' means. New `TaskDriver::with_config` is the constructor that takes it.")
52//! @yah:handoff("yah side: `crates/yah/kg-daemon/src/service.rs::open_task_store` now passes `AdoptLiveHosts { origins: [ORIGIN_TERMINAL] }`. Also replaced the magic string — new `kg_daemon::shell_vt::ORIGIN_TERMINAL` now backs the two live `origin == \"terminal\"` gates in shell_vt.rs plus the policy, so the VT-parsing gate and the tombstone-exemption gate cannot drift apart by a typo. The constant lives on the yah side, NOT in task-runs, precisely to keep the crate generic.")
53//! @yah:handoff("Also stamped at app/yah/desktop/src/terminal.rs:519 — the desktop-local PTY path (terminal_open_local's scrollback mint) owns its own PTYs, so those rows carry the desktop's pid. Without it the shell host's driver would tombstone a live desktop-local session on attach, which is the same bug pointing the other way.")
54//! @yah:handoff("PID REUSE is the honest weakness and is why the policy is opt-in and origin-narrowed. `kill(pid, 0)` (EPERM counts as alive — the process exists, it is just not ours to signal) can read a recycled pid as the original owner. The failure mode of a false 'alive' is one run left `Running` until something closes it; the false 'dead' this replaces kills a live session's status. Strictly the better direction for an interactive shell, and the exposure is bounded to origins the embedder opted in. Non-unix has no kill(2), so `host_process_alive` reports false there and the platform keeps the old behaviour rather than stranding runs forever.")
55//! @yah:handoff("SIX NEW TESTS, each pinned to a failure rather than a code path: a live-owner terminal run survives a new driver (the ticket's whole point); a run whose owner pid was spawned and reaped in-test IS tombstoned (a crashed host must not leave zombie tiles); origin-less and non-matching origins are tombstoned even with a live owner (an in-flight `cargo build` whose driver is gone has nobody left to record its exit); an unattributed row (pre-migration) is tombstoned; `TaskDriver::new` still tombstones unconditionally (no silent behaviour change for existing embedders); and `spawn_run` stamps this process — the policy is worthless if rows arrive unattributed.")
56//! @yah:verify("cd oss/qed && cargo test -p task-runs --lib  # 243/243, 6 new under driver::tests")
57//! @yah:verify("cargo test -p kg-daemon --lib shell_vt  # 9/9")
58//! @yah:verify("cargo test -p yah --lib r617  # 9/9")
59//! @yah:verify("Manual (needs a desktop rebuild): open a shell, run `sleep 300`, quit and relaunch the desktop — the run is still Running, not Lost")
60//! @yah:verify("sqlite3 .yah/db/task-runs.turso \"select id, origin, host_pid, status from runs where status='running';\"  # every live row names a pid that ps shows")
61//! @yah:gotcha("This is an oss/qed crate — changes land in-tree under oss/qed/crates/task-runs and flow outward via scripts/export-oss.sh. The seam was kept origin-agnostic (StaleRunPolicy decides on host_pid, takes origins as data); the one yah-ism, ORIGIN_TERMINAL, lives in crates/yah/kg-daemon/src/shell_vt.rs instead.")
62//! @yah:gotcha("`host_pid` is NOT on the wire. rpc::WireRunMeta does not carry it, so a client cannot ask 'is this run's owner alive'. Nothing needs it today — the policy runs entirely daemon-side — but R617-F7 should check whether reattaching tiles want it before adding a second liveness notion of their own.")
63//! @yah:gotcha("pid reuse can make a dead owner read alive, leaving a run `Running` with nobody driving it. Bounded on purpose (opt-in + origin-narrowed) and strictly safer than the false-dead it replaces, but it is a real edge: if zombie terminal rows ever accumulate, this is why.")
64//! @yah:gotcha("TaskRunMeta gained a required field, so every struct-literal construction site had to be updated (velveteen-exec x4, scryer, task-runs fixtures, kg-daemon fixtures, desktop/terminal.rs x2). A new construction site added by anyone else will fail to compile until they pick a value — which is the intended forcing function: a run with no recorded owner is a run the policy has to tombstone.")
65//!
66//! @yah:ticket(R617-B9, "Pre-existing: task-runs log_pipe_events_land_in_store never completes (233 pass / 1 fail)")
67//! @yah:status(review)
68//! @yah:assignee(agent:bundle-anthropic-ashguard)
69//! @yah:at(2026-07-22T19:50:25Z)
70//! @yah:phase(P1)
71//! @yah:parent(R617)
72//! @yah:handoff("Root cause: not the FIFO, not the PTY. The whole pipeline completed correctly every time (child wrote the JSON line, receiver drained it, reader hit EOF, child.wait returned 0) — but the lifecycle's terminal `store.update_status` returned `Sql(Busy(\"database is locked\"))` and run_lifecycle swallowed it with `let _ =`, so the run stayed Running forever and the 20s poll deadline blew. A live run has three concurrent turso writers (PTY chunk appends, shim-FIFO event appends, lifecycle status) on independent connections with no busy handling at all.")
73//! @yah:handoff("Fix in oss/qed/crates/task-runs/src/store.rs: (1) `conn()` now sets `busy_timeout(5s)` on every connection; (2) new `exec_retry()` wraps writes in an outer exponential-backoff retry on the `Busy`/`BusySnapshot` class, because turso caps its internal backoff and then hands `Busy` back; (3) insert_run / update_status / update_beholder_status / append_chunk / append_event all routed through it.")
74//! @yah:handoff("driver.rs run_lifecycle no longer swallows the terminal status write — a genuine failure after retries now prints `[yah task-runs] failed to record terminal status for run <id>`, matching the crate's existing eprintln convention.")
75//! @yah:handoff("New regression test store.rs::concurrent_writers_do_not_lose_the_terminal_status — two background tasks hammer append_chunk/append_event while update_status lands. Verified it has teeth: with busy_timeout and the retry disabled it fails 3/3 with the exact `Busy(\"database is locked\")`; with them it passes 5/5.")
76//! @yah:verify("cd oss/qed && cargo test -p task-runs --lib — 237 passed / 0 failed (was 235 pass / 1 fail)")
77//! @yah:verify("log_pipe_events_land_in_store run 8x sequentially: 8/8 green in ~0.58s each. Before the fix the same loop was 11/12 red at the 20s timeout.")
78//!
79//! @yah:ticket(R652-T6, "Login shell: when cmd is the resolved shell, exec it directly (not sh -c) with -l")
80//! @yah:at(2026-08-02T00:03:08Z)
81//! @yah:status(review)
82//! @yah:assignee(agent:bundle-ollama-cloud-boulder)
83//! @yah:phase(P1)
84//! @yah:parent(R652)
85//! @yah:handoff("Login shells now exec directly with -l instead of going through sh -c. SpawnOpts (oss/qed/crates/task-runs/src/driver.rs) gained `argv: Option<Vec<String>>`: when set, spawn_run builds the CommandBuilder from that argv verbatim instead of wrapping `cmd` in `sh -c`. camp-service task_run sets it to [resolved_shell, \"-l\"] whenever the request is a shell request.")
86//! @yah:handoff("Why an argv escape hatch rather than a `login_shell: bool` flag in the driver: task-runs is an oss/qed crate and has no business knowing what a login shell is. The caller names the exact process; the driver just execs it. This also made R652-T4 a two-line addition rather than a second flag.")
87//! @yah:handoff("Three things this fixes beyond .zprofile finally running. (1) `sh -c \"zsh -l\"` left an inert `sh` as the PTY's foreground process group leader, so job control misbehaved and signals went to the wrong process. (2) That same inert sh is what the foreground-pid cwd probe (R652-T2) would have reported for, so T2 could not have worked without this. (3) -l is now a real argv element instead of text inside a shell string, so no quoting layer can eat it.")
88//! @yah:handoff("`cmd` is still what lands on TaskRunMeta.command, so a shell run reads back as \"$SHELL\" -- the rail label and the history re-run path both keep working. Beholder argv rewriting is bypassed when argv is set (the attach runs with BeholderSelect::None): the rewritten argv would be discarded on that path, so recording a `rewrite=...` that never happened would be a lie in the run metadata.")
89//! @yah:handoff("An empty argv falls back to the sh -c path rather than spawning nothing -- a caller bug should not become an exec of the empty string.")
90//! @yah:verify("cd oss/qed && cargo test -p task-runs --lib  # 246/246 green (3 new: explicit_argv_execs_the_program_directly, explicit_argv_still_records_the_requested_command, empty_argv_falls_back_to_the_shell_path)")
91//! @yah:verify("Manual (needs desktop rebuild): add `echo W289-login-test >> /tmp/w289.log` to ~/.zprofile, open a shell tile, confirm the file gets a line")
92//! @yah:gotcha("driver.rs is an oss/qed crate -- this lands in-tree under oss/qed/crates/task-runs and flows outward via scripts/export-oss.sh on the next release. SpawnOpts gained a field, but every in-tree construction site uses ..Default::default(), so nothing else needed touching.")
93
94use std::collections::HashMap;
95use std::io::Read;
96use std::path::PathBuf;
97use std::sync::{Arc, Mutex};
98use std::time::{Duration, SystemTime, UNIX_EPOCH};
99
100use portable_pty::{native_pty_system, CommandBuilder, PtySize};
101use thiserror::Error;
102use tokio::sync::{mpsc, oneshot};
103use tokio::task;
104
105use crate::beholders::{registry_with_user_beholders, BeholderSelect};
106use crate::store::{RunFilter, StoreError, TaskStore};
107use crate::types::{BeholderStatus, Initiator, OutputChunk, RunStatus, Stream, TaskRunId, TaskRunMeta};
108
109const DEFAULT_GRACE: Duration = Duration::from_secs(5);
110const READ_BUF_SIZE: usize = 4096;
111const SIGTERM: i32 = 15;
112const SIGKILL: i32 = 9;
113
114// ─── Error ────────────────────────────────────────────────────────────────────
115
116#[derive(Debug, Error)]
117pub enum DriverError {
118    #[error("store: {0}")]
119    Store(#[from] StoreError),
120    #[error("pty: {0}")]
121    Pty(String),
122    #[error("run not found: {0}")]
123    NotFound(String),
124    #[error("io: {0}")]
125    Io(#[from] std::io::Error),
126}
127
128// ─── SpawnOpts ────────────────────────────────────────────────────────────────
129
130/// Options for [`TaskDriver::spawn_run`].
131#[derive(Debug, Clone)]
132pub struct SpawnOpts {
133    pub cwd: PathBuf,
134    /// Env vars set on the child process (merged on top of the current env).
135    pub env: Vec<(String, String)>,
136    pub label: Option<String>,
137    pub initiator: Initiator,
138    /// PTY column count. Defaults to 80.
139    pub pty_cols: u16,
140    /// PTY row count. Defaults to 24.
141    pub pty_rows: u16,
142    /// Enable stdin relay via [`TaskDriver::send_stdin`].
143    pub stdin_enabled: bool,
144    /// Pin the run so the GC sweep does not drop its output during warm rolloff.
145    pub pin: bool,
146    /// Beholder attachment policy. Defaults to [`BeholderSelect::Auto`].
147    pub beholder_select: BeholderSelect,
148    /// `true` when a human-facing terminal tile is attached. Causes `Rewriter`
149    /// beholders to decline in `Auto` mode so the human sees unmodified output.
150    pub tty_attached: bool,
151    /// Create a side-channel FIFO and export `YAH_TASK_RUN` / `YAH_LOG_PIPE`
152    /// so Tier-2 shim libraries (yah-log-rust, @yah/log) can emit structured
153    /// events. Has no effect on non-Unix platforms. Defaults to `true`.
154    pub log_fd_enabled: bool,
155    /// Provenance tag stored on the run's `TaskRunMeta.origin` (e.g.
156    /// `Some("terminal")` for an interactive shell). `None` is an ordinary job.
157    pub origin: Option<String>,
158    /// Exec this argv directly instead of wrapping `cmd` in `sh -c`.
159    ///
160    /// The default `sh -c <cmd>` is right for a job — the caller wrote a
161    /// command line and expects a shell to parse it. It is wrong for an
162    /// *interactive shell*: `sh -c "zsh -l"` leaves an inert `sh` as the PTY's
163    /// foreground process group leader, so job control misbehaves, signals go
164    /// to the wrong process, and anything that reads the foreground pid (a
165    /// live-cwd probe, say) sees `sh` instead of the shell the operator is
166    /// typing into. Handing the exact argv here makes the shell itself the
167    /// child, which is also the only way to pass `-l` as a real argv element
168    /// so `.zprofile` / `.profile` actually run.
169    ///
170    /// `cmd` is still what gets recorded on `TaskRunMeta.command`, so the run
171    /// reads the way the caller asked for it. Beholder argv rewriting is
172    /// bypassed when this is set: the caller has already decided the exact
173    /// process to exec, and a recorded `rewrite=…` that didn't happen would be
174    /// a lie in the run metadata.
175    pub argv: Option<Vec<String>>,
176}
177
178impl Default for SpawnOpts {
179    fn default() -> Self {
180        Self {
181            cwd: std::env::current_dir().unwrap_or_else(|_| PathBuf::from("/")),
182            env: vec![],
183            label: None,
184            initiator: Initiator::Human { camp: "local".to_string() },
185            pty_cols: 80,
186            pty_rows: 24,
187            stdin_enabled: false,
188            pin: false,
189            beholder_select: BeholderSelect::Auto,
190            tty_attached: false,
191            log_fd_enabled: true,
192            origin: None,
193            argv: None,
194        }
195    }
196}
197
198// ─── Driver channels ─────────────────────────────────────────────────────────
199
200/// Optional side-channels a driver can publish to. Both are fire-and-forget:
201/// a closed receiver never stalls or fails a run.
202#[derive(Default)]
203pub struct DriverChannels {
204    /// Fires `(run_id, status)` after each run's lifecycle task writes the
205    /// terminal status. Drives completion listeners (e.g. a triage worker).
206    pub completion: Option<mpsc::UnboundedSender<(TaskRunId, RunStatus)>>,
207    /// Mirrors every PTY output chunk as it is captured, *before* any consumer
208    /// polls the store. Lets a host attach a live view (VT parser, log
209    /// forwarder) to a run without a read-back loop over the store.
210    ///
211    /// The driver deliberately stays ignorant of what the tap is for — the
212    /// chunk carries `run_id`, so the host decides which runs it cares about.
213    pub output: Option<mpsc::UnboundedSender<OutputChunk>>,
214}
215
216// ─── Stale-run policy ────────────────────────────────────────────────────────
217
218/// What a freshly-constructed [`TaskDriver`] does with `Running` rows it finds
219/// already in the store.
220///
221/// The historical rule — tombstone every one of them — bakes in an assumption
222/// that stops being true the moment a second process attaches to the same
223/// store: that any `Running` row must be a corpse from *this* process's
224/// predecessor. When two processes share a store, a driver starting up in one
225/// will happily mark the other's live runs `Lost`, and the run keeps producing
226/// output under a status that says it is dead.
227///
228/// The policy is deliberately origin-agnostic in its mechanism — it decides on
229/// **who owns the run** ([`TaskRunMeta::host_pid`]) — and takes the origin list
230/// as data, so an embedder names the runs it wants exempted without this crate
231/// knowing what any of them mean.
232#[derive(Debug, Clone, Default, PartialEq, Eq)]
233pub enum StaleRunPolicy {
234    /// Tombstone every leftover `Running` run as `Lost`.
235    ///
236    /// Correct, and the default, whenever this process is the only writer:
237    /// a run whose driver is gone has no one left to notice it exit.
238    #[default]
239    LostOnDisappear,
240    /// Spare runs whose recorded owner process is still alive.
241    ///
242    /// A leftover run is tombstoned only when its `host_pid` is absent (owner
243    /// unknown — a row from before the column existed) or names a process that
244    /// no longer exists. Anything else belongs to a live peer and is left
245    /// `Running` for that peer to finish.
246    ///
247    /// `origins` narrows the exemption to runs whose
248    /// [`TaskRunMeta::origin`] is in the list; empty means every origin
249    /// qualifies. A run with no origin never matches a non-empty list.
250    AdoptLiveHosts { origins: Vec<String> },
251}
252
253impl StaleRunPolicy {
254    /// Whether `meta` should be tombstoned `Lost` at driver construction.
255    fn tombstones(&self, meta: &TaskRunMeta) -> bool {
256        match self {
257            StaleRunPolicy::LostOnDisappear => true,
258            StaleRunPolicy::AdoptLiveHosts { origins } => {
259                let exempt_origin = origins.is_empty()
260                    || meta
261                        .origin
262                        .as_deref()
263                        .is_some_and(|o| origins.iter().any(|want| want == o));
264                if !exempt_origin {
265                    return true;
266                }
267                match meta.host_pid {
268                    Some(pid) => !host_process_alive(pid),
269                    None => true,
270                }
271            }
272        }
273    }
274}
275
276/// Is a process with this pid still around?
277///
278/// `kill(pid, 0)` is the portable liveness probe: it performs the permission
279/// check and existence lookup without delivering anything. `EPERM` counts as
280/// alive — the process exists, it just is not ours to signal.
281///
282/// Pid reuse can make a dead owner read as alive. That is why
283/// [`StaleRunPolicy::AdoptLiveHosts`] is opt-in and origin-narrowed: the cost
284/// of a false "alive" is one run left `Running` until something closes it,
285/// which is strictly better for an interactive session than the false "dead"
286/// this replaces — which kills a *live* session's status.
287#[cfg(unix)]
288fn host_process_alive(pid: u32) -> bool {
289    if pid == 0 {
290        return false;
291    }
292    if pid == std::process::id() {
293        return true;
294    }
295    // SAFETY: `kill` with signal 0 delivers nothing; it only reports whether
296    // the pid exists and is signallable.
297    let rc = unsafe { libc::kill(pid as libc::pid_t, 0) };
298    rc == 0 || std::io::Error::last_os_error().raw_os_error() == Some(libc::EPERM)
299}
300
301/// No `kill(2)` off Unix. Reporting every owner dead keeps the historical
302/// Lost-on-disappear behaviour rather than stranding runs `Running` forever.
303#[cfg(not(unix))]
304fn host_process_alive(_pid: u32) -> bool {
305    false
306}
307
308// ─── Internal run-control handle ─────────────────────────────────────────────
309
310struct RunControl {
311    kill_tx: mpsc::Sender<KillRequest>,
312    stdin_tx: Option<mpsc::Sender<Vec<u8>>>,
313    /// Shared with the lifecycle task, which holds the same `Arc` so the PTY fd
314    /// outlives `child.wait()`. `MasterPty::resize` takes `&self`, so a mutex is
315    /// enough to make the `Box<dyn MasterPty + Send>` `Sync` across the two.
316    master: Arc<Mutex<Box<dyn portable_pty::MasterPty + Send>>>,
317}
318
319#[derive(Debug)]
320struct KillRequest {
321    signal: i32,
322}
323
324// ─── ShimRecord ───────────────────────────────────────────────────────────────
325
326/// One JSON-line record emitted by a Tier-2 shim to the side-channel FIFO.
327///
328/// The shim (Rust `yah-log` layer or TS `@yah/log` pino transport) writes one
329/// of these per log call. Unknown keys inside `fields` pass through unchanged.
330#[cfg(unix)]
331#[derive(serde::Deserialize)]
332struct ShimRecord {
333    level: String,
334    target: String,
335    msg: String,
336    #[serde(default)]
337    fields: serde_json::Value,
338    /// Shim library name, e.g. `"yah-log-rust"`. Populates
339    /// [`EventSource::Shim::lib`].
340    #[serde(rename = "_lib", default)]
341    lib: Option<String>,
342    /// Shim library version string.
343    #[serde(rename = "_lib_ver", default)]
344    lib_version: Option<String>,
345}
346
347// ─── FdCloser ─────────────────────────────────────────────────────────────────
348
349/// RAII wrapper that closes a raw fd on drop.
350///
351/// Used to hold the write end of the log FIFO open until the lifecycle task
352/// completes. Dropping it signals EOF to the receiver thread.
353#[cfg(unix)]
354struct FdCloser(libc::c_int);
355
356#[cfg(unix)]
357impl Drop for FdCloser {
358    fn drop(&mut self) {
359        unsafe { libc::close(self.0) };
360    }
361}
362
363// SAFETY: a raw fd number is an integer; closing it from any thread is safe
364// provided we never duplicate ownership (enforced by move semantics here).
365#[cfg(unix)]
366unsafe impl Send for FdCloser {}
367
368// ─── TaskDriver ───────────────────────────────────────────────────────────────
369
370/// Manages in-flight task runs for a single camp.
371///
372/// Wrap in `Arc` to share across tasks; internal state is mutex-protected.
373pub struct TaskDriver {
374    store: Arc<TaskStore>,
375    active: Arc<Mutex<HashMap<String, RunControl>>>,
376    /// Side-channels published to by every run this driver owns.
377    channels: DriverChannels,
378}
379
380impl TaskDriver {
381    /// Create a driver backed by `store`, with no side-channels.
382    ///
383    /// Immediately scans the store for `Running` runs left over from a prior
384    /// daemon process and marks them `Lost` ("Lost-on-disappear").
385    pub async fn new(store: Arc<TaskStore>) -> Result<Self, DriverError> {
386        Self::with_channels(store, DriverChannels::default()).await
387    }
388
389    /// Like `new` but wires the optional [`DriverChannels`] side-channels
390    /// (completion notifications, live output tap).
391    pub async fn with_channels(
392        store: Arc<TaskStore>,
393        channels: DriverChannels,
394    ) -> Result<Self, DriverError> {
395        Self::with_config(store, channels, StaleRunPolicy::default()).await
396    }
397
398    /// Full constructor: side-channels plus the [`StaleRunPolicy`] applied to
399    /// `Running` rows already in the store.
400    ///
401    /// R617-F6 — annotation in this file's header. Splitting the sweep out of
402    /// the constructor's fixed behaviour is what lets a store be shared: a
403    /// process that is not the run's owner can now attach without declaring
404    /// the owner's live work dead.
405    pub async fn with_config(
406        store: Arc<TaskStore>,
407        channels: DriverChannels,
408        stale_policy: StaleRunPolicy,
409    ) -> Result<Self, DriverError> {
410        let stale = store
411            .list_runs(&RunFilter {
412                status: Some("running".to_string()),
413                ..Default::default()
414            })
415            .await?;
416        for meta in stale {
417            if !stale_policy.tombstones(&meta) {
418                continue;
419            }
420            store
421                .update_status(
422                    &meta.id,
423                    &RunStatus::Lost {
424                        reason: "daemon restarted while run was in-flight".to_string(),
425                    },
426                )
427                .await?;
428        }
429        Ok(Self {
430            store,
431            active: Arc::new(Mutex::new(HashMap::new())),
432            channels,
433        })
434    }
435
436    /// Spawn `cmd` in a PTY and start capturing its output. Returns immediately
437    /// with the new [`TaskRunId`].
438    ///
439    /// A beholder is selected via `opts.beholder_select` (default `Auto`). When
440    /// a `Rewriter` beholder matches, its `adjust_argv` is applied to the
441    /// command before spawning and the diff is recorded on `beholder_status`.
442    /// When `opts.tty_attached` is `true`, `Rewriter` beholders decline in
443    /// `Auto` mode to preserve human-readable output.
444    ///
445    /// Output is written to the store as `Stream::Stdout` chunks (the PTY
446    /// kernel merges stdout and stderr). Signal handling and status updates
447    /// run in background tasks.
448    pub async fn spawn_run(&self, cmd: &str, opts: SpawnOpts) -> Result<TaskRunId, DriverError> {
449        let id = TaskRunId::new();
450        let started_at = unix_now_secs();
451        let started_at_ms: u64 = started_at.saturating_mul(1000);
452
453        // Attach a beholder (may rewrite argv and produce structured events).
454        // Resolve user drop-in directory: $YAH_BEHOLDERS_DIR or $HOME/.yah/beholders.
455        let user_dir = std::env::var_os("YAH_BEHOLDERS_DIR")
456            .map(std::path::PathBuf::from)
457            .or_else(|| {
458                std::env::var_os("HOME")
459                    .map(|h| std::path::PathBuf::from(h).join(".yah/beholders"))
460            });
461        let registry = registry_with_user_beholders(user_dir.as_deref());
462        /* An explicit argv means the caller already chose the exact process
463           (an interactive login shell, say). Selecting a beholder there would
464           either do nothing — the rewritten argv is discarded on that path —
465           or record a rewrite that never happened, so we opt out honestly
466           instead. */
467        let select = if opts.argv.is_some() {
468            &BeholderSelect::None
469        } else {
470            &opts.beholder_select
471        };
472        let attach = registry.attach(cmd, select, opts.tty_attached);
473        // Reconstruct the command from argv ONLY when a beholder actually
474        // rewrote it. `AttachResult.argv` is always populated — it is
475        // `resolve_argv(cmd)` even when nothing attached — so joining it
476        // unconditionally ran every run's command through a whitespace
477        // normalization nobody asked for: runs of spaces collapse and embedded
478        // newlines become spaces, which is silent corruption for a heredoc or
479        // any multi-line line. The caller's bytes go to the shell untouched
480        // unless a rewrite is the whole point.
481        let effective_cmd = match &attach.status.rewrite_added {
482            Some(added) if !added.is_empty() && !attach.argv.is_empty() => attach.argv.join(" "),
483            _ => cmd.to_string(),
484        };
485
486        self.store.insert_run(&TaskRunMeta {
487            id: id.clone(),
488            command: cmd.to_string(),
489            cwd: opts.cwd.clone(),
490            env: opts.env.clone(),
491            started_at,
492            status: RunStatus::Running,
493            label: opts.label.clone(),
494            initiator: opts.initiator.clone(),
495            beholder_status: Some(attach.status),
496            pinned: opts.pin,
497            origin: opts.origin.clone(),
498            /* R617-F6: stamp the OWNER, before the child exists. Written at
499               insert rather than after spawn so a crash between the two still
500               leaves the row attributable — an unattributed `Running` row is
501               exactly what the conservative arm of `StaleRunPolicy` has to
502               tombstone. */
503            host_pid: Some(std::process::id()),
504        }).await?;
505
506        // Open PTY pair.
507        let pty_sys = native_pty_system();
508        let pair = pty_sys
509            .openpty(PtySize {
510                rows: opts.pty_rows,
511                cols: opts.pty_cols,
512                pixel_width: 0,
513                pixel_height: 0,
514            })
515            .map_err(|e| DriverError::Pty(e.to_string()))?;
516
517        // Clone reader before spawning so the fd is ready immediately.
518        let pty_reader = pair
519            .master
520            .try_clone_reader()
521            .map_err(|e| DriverError::Pty(e.to_string()))?;
522
523        // Optional stdin relay: take the writer before spawning the child.
524        let stdin_tx: Option<mpsc::Sender<Vec<u8>>> = if opts.stdin_enabled {
525            let mut writer = pair
526                .master
527                .take_writer()
528                .map_err(|e| DriverError::Pty(e.to_string()))?;
529            let (tx, mut rx) = mpsc::channel::<Vec<u8>>(64);
530            task::spawn(async move {
531                use std::io::Write;
532                while let Some(bytes) = rx.recv().await {
533                    let _ = writer.write_all(&bytes);
534                    let _ = writer.flush();
535                }
536            });
537            Some(tx)
538        } else {
539            None
540        };
541
542        // ── Side-channel log FIFO (Tier 2 / yah-log shims) ──────────────────
543        //
544        // Create a named pipe (FIFO) so child processes can write structured
545        // events without touching stdout/stderr. We export its path via
546        // YAH_LOG_PIPE; no fd inheritance is involved, so portable-pty's
547        // close_random_fds() pre_exec hook doesn't interfere.
548        //
549        // The parent opens the FIFO twice:
550        //   rfd — O_RDONLY|O_NONBLOCK, then cleared to blocking → read events
551        //   wfd — O_WRONLY (wrapped in FdCloser) → keeps the FIFO alive until
552        //          the lifecycle task drops it (after run completion), producing
553        //          EOF for the receiver thread.
554        #[cfg(unix)]
555        let log_fifo: Option<(libc::c_int, FdCloser, std::path::PathBuf)> = if opts.log_fd_enabled {
556            let fifo_path = std::env::temp_dir().join(format!("yah-log-{}.fifo", id));
557            let path_cstr = match std::ffi::CString::new(fifo_path.to_string_lossy().as_bytes()) {
558                Ok(s) => s,
559                Err(_) => {
560                    // Path contained a nul byte — extremely unlikely; skip FIFO.
561                    return Err(DriverError::Io(std::io::Error::new(
562                        std::io::ErrorKind::InvalidInput,
563                        "log FIFO path contained nul byte",
564                    )));
565                }
566            };
567            let mkfifo_ret = unsafe { libc::mkfifo(path_cstr.as_ptr(), 0o600) };
568            if mkfifo_ret != 0 {
569                None // FIFO creation failed; continue without side-channel
570            } else {
571                // Open read end without blocking (no writer yet).
572                let rfd = unsafe {
573                    libc::open(path_cstr.as_ptr(), libc::O_RDONLY | libc::O_NONBLOCK)
574                };
575                if rfd < 0 {
576                    let _ = unsafe { libc::unlink(path_cstr.as_ptr()) };
577                    None
578                } else {
579                    // Switch read end to blocking so reads yield proper data.
580                    unsafe { libc::fcntl(rfd, libc::F_SETFL, 0) };
581                    // Open write end — this succeeds immediately because rfd is open.
582                    let wfd = unsafe {
583                        libc::open(path_cstr.as_ptr(), libc::O_WRONLY)
584                    };
585                    if wfd < 0 {
586                        unsafe { libc::close(rfd) };
587                        let _ = unsafe { libc::unlink(path_cstr.as_ptr()) };
588                        None
589                    } else {
590                        Some((rfd, FdCloser(wfd), fifo_path))
591                    }
592                }
593            }
594        } else {
595            None
596        };
597
598        // Build and spawn the child inside the slave. An explicit argv execs
599        // that program directly; otherwise the command line goes through `sh`
600        // so the caller's quoting, pipes and redirections mean what they say.
601        let mut cb = match opts.argv.as_deref() {
602            Some([program, args @ ..]) => {
603                let mut cb = CommandBuilder::new(program);
604                cb.args(args);
605                cb
606            }
607            // An empty argv is a caller bug, not a request for an empty exec —
608            // fall back to the shell path rather than spawning nothing.
609            _ => {
610                let mut cb = CommandBuilder::new("sh");
611                cb.args(["-c", &effective_cmd]);
612                cb
613            }
614        };
615        cb.cwd(&opts.cwd);
616        for (k, v) in &opts.env {
617            cb.env(k, v);
618        }
619        cb.env("TERM", "xterm-256color");
620
621        // Export YAH_TASK_RUN and YAH_LOG_PIPE if the FIFO was created.
622        #[cfg(unix)]
623        if let Some((_, _, ref fifo_path)) = log_fifo {
624            cb.env("YAH_TASK_RUN", id.to_string());
625            cb.env("YAH_LOG_PIPE", fifo_path.to_string_lossy().as_ref());
626        }
627
628        let child = pair
629            .slave
630            .spawn_command(cb)
631            .map_err(|e| DriverError::Pty(e.to_string()))?;
632        // Drop the parent's slave handle so EOF propagates once the child exits.
633        drop(pair.slave);
634
635        // Share the master between the lifecycle task (which must outlive
636        // `child.wait()` so the fd stays open) and `resize_run`.
637        let master: Arc<Mutex<Box<dyn portable_pty::MasterPty + Send>>> =
638            Arc::new(Mutex::new(pair.master));
639
640        let pid = child.process_id().unwrap_or(0);
641
642        // ── FIFO: launch receiver thread; pass write-end holder to lifecycle ──
643        //
644        // The receiver thread reads until EOF. EOF arrives when ALL write-end
645        // holders close: the child's own writers (when it exits) plus the
646        // FdCloser we hand to the lifecycle task (which drops it after writing
647        // the terminal RunStatus). Events written before the last close are
648        // still drained by the receiver thread before it exits.
649        #[cfg(unix)]
650        let log_wfd_holder: Option<FdCloser> = if let Some((rfd, wfd, fifo_path)) = log_fifo {
651            let store_log = Arc::clone(&self.store);
652            let id_log = id.clone();
653            let rt = tokio::runtime::Handle::current();
654            // spawn_blocking: lets the runtime track this thread so the
655            // Handle::block_on calls inside have a worker to drive futures.
656            tokio::task::spawn_blocking(move || {
657                run_log_receiver(rt, store_log, id_log, rfd, fifo_path, started_at_ms);
658            });
659            Some(wfd)
660        } else {
661            None
662        };
663
664        // Channels.
665        let (kill_tx, kill_rx) = mpsc::channel::<KillRequest>(4);
666        let (reader_done_tx, reader_done_rx) = oneshot::channel::<()>();
667
668        // Reader thread: PTY output → store chunks → beholder events.
669        // Runs on a dedicated OS thread because PTY reads are blocking.
670        {
671            let store_r = Arc::clone(&self.store);
672            let id_r = id.clone();
673            let mut beholder = attach.beholder;
674            let output_tx = self.channels.output.clone();
675            let rt = tokio::runtime::Handle::current();
676            tokio::task::spawn_blocking(move || {
677                let mut buf = [0u8; READ_BUF_SIZE];
678                let mut reader = pty_reader;
679                loop {
680                    match reader.read(&mut buf) {
681                        Ok(0) | Err(_) => break,
682                        Ok(n) => {
683                            let offset = elapsed_ms(started_at_ms);
684                            let append_res = rt.block_on(store_r.append_chunk(
685                                &id_r,
686                                offset,
687                                Stream::Stdout,
688                                &buf[..n],
689                            ));
690                            if let Ok(seq) = append_res {
691                                /* Both the tap and the beholder want the same
692                                   owned chunk; build it once, and only when
693                                   someone is listening. */
694                                let chunk = (output_tx.is_some() || beholder.is_some()).then(|| {
695                                    OutputChunk {
696                                        run_id: id_r.clone(),
697                                        seq,
698                                        offset_ms: offset,
699                                        stream: Stream::Stdout,
700                                        bytes: buf[..n].to_vec(),
701                                    }
702                                });
703                                /* Tap first: it feeds live views, where latency
704                                   is visible to a human. Send failure means the
705                                   host dropped its receiver — never fatal. */
706                                if let (Some(tx), Some(c)) = (&output_tx, &chunk) {
707                                    let _ = tx.send(c.clone());
708                                }
709                                let mut detach_beholder = false;
710                                if let (Some(b), Some(chunk)) = (beholder.as_mut(), &chunk) {
711                                    for ev in b.parse_chunk(chunk) {
712                                        let _ = rt.block_on(store_r.append_event(
713                                            &ev.run_id,
714                                            ev.offset_ms,
715                                            ev.level,
716                                            &ev.target,
717                                            &ev.msg,
718                                            &ev.fields,
719                                            ev.anchor.as_ref().map(|a| a.seq),
720                                            &ev.source,
721                                        ));
722                                    }
723                                    if let Some(reason) = b.unknown_format_reason() {
724                                        let new_status = BeholderStatus::unknown_format_with_reason(
725                                            b.name(),
726                                            reason,
727                                        );
728                                        let _ = rt.block_on(
729                                            store_r.update_beholder_status(&id_r, &new_status),
730                                        );
731                                        detach_beholder = true;
732                                    }
733                                }
734                                if detach_beholder {
735                                    beholder = None;
736                                }
737                            }
738                        }
739                    }
740                }
741                if let Some(ref mut b) = beholder {
742                    let final_offset = elapsed_ms(started_at_ms);
743                    for ev in b.on_done(&id_r, final_offset) {
744                        let _ = rt.block_on(store_r.append_event(
745                            &ev.run_id,
746                            ev.offset_ms,
747                            ev.level,
748                            &ev.target,
749                            &ev.msg,
750                            &ev.fields,
751                            ev.anchor.as_ref().map(|a| a.seq),
752                            &ev.source,
753                        ));
754                    }
755                    if let Some(reason) = b.unknown_format_reason() {
756                        let new_status = BeholderStatus::unknown_format_with_reason(b.name(), reason);
757                        let _ = rt.block_on(store_r.update_beholder_status(&id_r, &new_status));
758                    }
759                }
760                let _ = reader_done_tx.send(());
761            });
762        }
763
764        // Lifecycle task: monitor kill requests, wait for exit, update status.
765        // The task also holds the log FIFO write-end closer (if any) so that
766        // EOF propagates to the receiver thread after RunStatus is written.
767        {
768            let store_l = Arc::clone(&self.store);
769            let active_l = Arc::clone(&self.active);
770            let id_l = id.clone();
771            let master_l = Arc::clone(&master);
772            let completion_tx_l = self.channels.completion.clone();
773            #[cfg(unix)]
774            let wfd_l = log_wfd_holder;
775            task::spawn(async move {
776                run_lifecycle(
777                    store_l,
778                    active_l,
779                    id_l,
780                    pid,
781                    child,
782                    master_l,
783                    kill_rx,
784                    reader_done_rx,
785                    completion_tx_l,
786                    #[cfg(unix)]
787                    wfd_l,
788                )
789                .await;
790            });
791        }
792
793        self.active
794            .lock()
795            .unwrap()
796            .insert(id.to_string(), RunControl { kill_tx, stdin_tx, master });
797
798        Ok(id)
799    }
800
801    /// Resize a running task's PTY and deliver `SIGWINCH` to the foreground
802    /// process group (portable-pty's `resize` does the ioctl, which is what
803    /// signals the child).
804    ///
805    /// Returns `DriverError::NotFound` when the run is not active on this
806    /// driver instance — the same contract as [`TaskDriver::send_stdin`].
807    pub async fn resize_run(
808        &self,
809        id: &TaskRunId,
810        cols: u16,
811        rows: u16,
812    ) -> Result<(), DriverError> {
813        let master = self
814            .active
815            .lock()
816            .unwrap()
817            .get(&id.to_string())
818            .map(|c| Arc::clone(&c.master));
819
820        match master {
821            Some(m) => {
822                let size = PtySize { rows, cols, pixel_width: 0, pixel_height: 0 };
823                m.lock()
824                    .unwrap()
825                    .resize(size)
826                    .map_err(|e| DriverError::Pty(e.to_string()))
827            }
828            None => Err(DriverError::NotFound(id.to_string())),
829        }
830    }
831
832    /// The pid of the run's *foreground* process — the leader of the process
833    /// group the PTY currently gives the keyboard to.
834    ///
835    /// For a shell tile that is the shell itself while it sits at a prompt,
836    /// and the command the operator is running while one is in flight. That
837    /// distinction is the whole point: asking the spawned child would report
838    /// the shell forever, so anything derived from this pid (a live cwd probe,
839    /// a "what is this pane doing" label) would answer for the wrong process.
840    ///
841    /// `None` when the run is not active on this driver instance, or when the
842    /// platform has no notion of a foreground process group.
843    pub fn foreground_pid(&self, id: &TaskRunId) -> Option<u32> {
844        let master = self
845            .active
846            .lock()
847            .unwrap()
848            .get(&id.to_string())
849            .map(|c| Arc::clone(&c.master))?;
850        #[cfg(unix)]
851        {
852            let pid = master.lock().unwrap().process_group_leader()?;
853            u32::try_from(pid).ok()
854        }
855        #[cfg(not(unix))]
856        {
857            let _ = master;
858            None
859        }
860    }
861
862    /// Send `signal` to a running task. Defaults to SIGTERM (15).
863    ///
864    /// For SIGTERM, the driver waits up to 5 seconds for the process to exit
865    /// before escalating to SIGKILL. Returns `DriverError::NotFound` if the
866    /// run is not active (already exited or launched on a different driver
867    /// instance).
868    pub async fn kill_run(&self, id: &TaskRunId, signal: Option<i32>) -> Result<(), DriverError> {
869        let kill_tx = self
870            .active
871            .lock()
872            .unwrap()
873            .get(&id.to_string())
874            .map(|c| c.kill_tx.clone());
875
876        match kill_tx {
877            Some(tx) => tx
878                .send(KillRequest { signal: signal.unwrap_or(SIGTERM) })
879                .await
880                .map_err(|_| DriverError::NotFound(id.to_string())),
881            None => Err(DriverError::NotFound(id.to_string())),
882        }
883    }
884
885    /// Write bytes to the stdin of a running task (requires `stdin_enabled`).
886    pub async fn send_stdin(&self, id: &TaskRunId, bytes: Vec<u8>) -> Result<(), DriverError> {
887        let stdin_tx = self
888            .active
889            .lock()
890            .unwrap()
891            .get(&id.to_string())
892            .and_then(|c| c.stdin_tx.clone());
893
894        match stdin_tx {
895            Some(tx) => tx
896                .send(bytes)
897                .await
898                .map_err(|_| DriverError::NotFound(id.to_string())),
899            None => Err(DriverError::NotFound(id.to_string())),
900        }
901    }
902}
903
904// ─── Log fd receiver ─────────────────────────────────────────────────────────
905
906/// Read JSON-lines from the side-channel FIFO read end and store them as
907/// [`EventSource::Shim`] events.
908///
909/// Runs on a dedicated OS thread; exits when the read end sees EOF. EOF
910/// arrives after both the child process AND the lifecycle task have closed
911/// their write ends of the FIFO. The FIFO file is deleted on exit.
912#[cfg(unix)]
913fn run_log_receiver(
914    rt: tokio::runtime::Handle,
915    store: Arc<TaskStore>,
916    run_id: TaskRunId,
917    read_fd: libc::c_int,
918    fifo_path: std::path::PathBuf,
919    started_at_ms: u64,
920) {
921    use std::io::BufRead;
922    use std::os::unix::io::FromRawFd;
923
924    // SAFETY: `read_fd` is a valid, open FIFO fd handed exclusively to this
925    // thread. `File` takes ownership and closes the fd on drop.
926    let file = unsafe { std::fs::File::from_raw_fd(read_fd) };
927    let reader = std::io::BufReader::new(file);
928
929    for line in reader.lines() {
930        let line = match line {
931            Ok(l) => l,
932            Err(_) => break,
933        };
934        let trimmed = line.trim();
935        if trimmed.is_empty() {
936            continue;
937        }
938        let rec: ShimRecord = match serde_json::from_str(trimmed) {
939            Ok(r) => r,
940            Err(_) => continue, // skip malformed lines silently
941        };
942        let level = rec.level.parse::<crate::types::Level>().unwrap_or(crate::types::Level::Info);
943        let source = crate::types::EventSource::Shim {
944            lib: rec.lib.unwrap_or_else(|| "unknown".to_string()),
945            version: rec.lib_version.unwrap_or_else(|| "0.0.0".to_string()),
946        };
947        let fields = if rec.fields.is_object() {
948            rec.fields
949        } else {
950            serde_json::Value::Object(Default::default())
951        };
952        let offset = elapsed_ms(started_at_ms);
953        let _ = rt.block_on(store.append_event(
954            &run_id,
955            offset,
956            level,
957            &rec.target,
958            &rec.msg,
959            &fields,
960            None,
961            &source,
962        ));
963    }
964
965    // Clean up the FIFO file now that the receiver has drained.
966    let _ = std::fs::remove_file(&fifo_path);
967}
968
969// ─── Lifecycle task ───────────────────────────────────────────────────────────
970
971async fn run_lifecycle(
972    store: Arc<TaskStore>,
973    active: Arc<Mutex<HashMap<String, RunControl>>>,
974    id: TaskRunId,
975    pid: u32,
976    child: Box<dyn portable_pty::Child + Send>,
977    master: Arc<Mutex<Box<dyn portable_pty::MasterPty + Send>>>,
978    mut kill_rx: mpsc::Receiver<KillRequest>,
979    reader_done_rx: oneshot::Receiver<()>,
980    completion_tx: Option<tokio::sync::mpsc::UnboundedSender<(TaskRunId, RunStatus)>>,
981    // Holds the write end of the log FIFO open until this task completes.
982    // Dropping it produces EOF for the receiver thread, which happens after
983    // the terminal RunStatus is written below.
984    #[cfg(unix)]
985    _log_wfd: Option<FdCloser>,
986) {
987    // Pin the reader-done future so it can be polled by reference in
988    // nested select! arms without consuming ownership.
989    let reader_done = async { reader_done_rx.await.ok(); };
990    tokio::pin!(reader_done);
991
992    let sent_signal: Option<i32>;
993
994    tokio::select! {
995        req = kill_rx.recv() => {
996            match req {
997                Some(KillRequest { signal }) => {
998                    send_unix_signal(pid, signal);
999                    if signal == SIGKILL {
1000                        sent_signal = Some(SIGKILL);
1001                    } else {
1002                        // Grace period: give the process a chance to exit cleanly.
1003                        tokio::select! {
1004                            _ = &mut reader_done => {
1005                                // Exited within grace — no SIGKILL needed.
1006                                sent_signal = Some(signal);
1007                            }
1008                            _ = tokio::time::sleep(DEFAULT_GRACE) => {
1009                                // Grace expired — escalate.
1010                                send_unix_signal(pid, SIGKILL);
1011                                sent_signal = Some(SIGKILL);
1012                            }
1013                        }
1014                    }
1015                }
1016                // kill_tx dropped (driver shutting down) — force kill.
1017                None => {
1018                    send_unix_signal(pid, SIGKILL);
1019                    sent_signal = Some(SIGKILL);
1020                }
1021            }
1022        }
1023        _ = &mut reader_done => {
1024            sent_signal = None;
1025        }
1026    }
1027
1028    // Reap the child (blocking) on a dedicated thread-pool slot.
1029    // Move our master handle in here so the PTY fd outlives the wait. The
1030    // matching `RunControl` (removed from `active` below) holds the other
1031    // `Arc`, so the fd actually closes once both are gone.
1032    let exit_code = task::spawn_blocking(move || {
1033        let mut c = child;
1034        let _m = master; // dropped after wait() returns
1035        c.wait().ok().map(|s| s.exit_code())
1036    })
1037    .await
1038    .ok()
1039    .flatten();
1040
1041    let ended_at = unix_now_secs();
1042    let status = match sent_signal {
1043        Some(sig) => RunStatus::Killed { signal: sig, ended_at },
1044        None => match exit_code {
1045            Some(code) => RunStatus::Done { exit_code: code as i32, ended_at },
1046            None => RunStatus::Lost {
1047                reason: "process exited without an exit code".to_string(),
1048            },
1049        },
1050    };
1051
1052    /* Losing this write is not cosmetic: the run stays `Running` in the store
1053       forever and every reader — tail loops, the terminal UI, the next
1054       daemon's Lost-on-disappear sweep — believes a dead process is alive.
1055       `update_status` already retries through lock contention, so a failure
1056       here is terminal and worth saying out loud. */
1057    if let Err(e) = store.update_status(&id, &status).await {
1058        eprintln!("[yah task-runs] failed to record terminal status for run {id}: {e}");
1059    }
1060    if let Some(ref tx) = completion_tx {
1061        let _ = tx.send((id.clone(), status));
1062    }
1063    active.lock().unwrap().remove(&id.to_string());
1064}
1065
1066// ─── Helpers ──────────────────────────────────────────────────────────────────
1067
1068fn send_unix_signal(pid: u32, signal: i32) {
1069    #[cfg(unix)]
1070    unsafe {
1071        libc::kill(pid as libc::pid_t, signal);
1072    }
1073    // On non-Unix platforms signal delivery is not implemented here.
1074}
1075
1076fn unix_now_secs() -> u64 {
1077    SystemTime::now()
1078        .duration_since(UNIX_EPOCH)
1079        .unwrap_or_default()
1080        .as_secs()
1081}
1082
1083fn elapsed_ms(started_at_ms: u64) -> u32 {
1084    let now_ms = SystemTime::now()
1085        .duration_since(UNIX_EPOCH)
1086        .unwrap_or_default()
1087        .as_millis() as u64;
1088    now_ms.saturating_sub(started_at_ms).min(u32::MAX as u64) as u32
1089}
1090
1091// ─── Tests ────────────────────────────────────────────────────────────────────
1092
1093#[cfg(test)]
1094mod tests {
1095    use super::*;
1096    use crate::store::ChunkFilter;
1097
1098    async fn open_store(dir: &tempfile::TempDir) -> Arc<TaskStore> {
1099        Arc::new(TaskStore::open(&dir.path().join("tr.turso")).await.unwrap())
1100    }
1101
1102    // ── Lost-on-disappear (pure store, no PTY) ────────────────────────────────
1103
1104    #[tokio::test]
1105    async fn lost_on_disappear_marks_stale_running_runs() {
1106        let dir = tempfile::tempdir().unwrap();
1107        let store = open_store(&dir).await;
1108
1109        // Simulate a run left in "Running" state by a prior daemon.
1110        let stale_id = TaskRunId::new();
1111        store
1112            .insert_run(&TaskRunMeta {
1113                id: stale_id.clone(),
1114                command: "sleep 9999".to_string(),
1115                cwd: "/tmp".into(),
1116                env: vec![],
1117                started_at: unix_now_secs() - 60,
1118                status: RunStatus::Running,
1119                label: None,
1120                initiator: Initiator::Human { camp: "test".to_string() },
1121                beholder_status: None,
1122                pinned: false,
1123                origin: None,
1124                host_pid: None,
1125            })
1126            .await
1127            .unwrap();
1128
1129        // Creating a new driver must mark stale runs Lost.
1130        let _driver = TaskDriver::new(Arc::clone(&store)).await.unwrap();
1131
1132        let meta = store.get_run(&stale_id).await.unwrap().unwrap();
1133        assert!(
1134            matches!(meta.status, RunStatus::Lost { .. }),
1135            "stale run should be Lost, got {:?}",
1136            meta.status
1137        );
1138    }
1139
1140    // ── Stale-run policy (R617-F6) ───────────────────────────────────────────
1141
1142    /// Plant a `Running` row as if some other process had spawned it.
1143    async fn plant_running(
1144        store: &Arc<TaskStore>,
1145        origin: Option<&str>,
1146        host_pid: Option<u32>,
1147    ) -> TaskRunId {
1148        let id = TaskRunId::new();
1149        store
1150            .insert_run(&TaskRunMeta {
1151                id: id.clone(),
1152                command: "sleep 9999".to_string(),
1153                cwd: "/tmp".into(),
1154                env: vec![],
1155                started_at: unix_now_secs() - 60,
1156                status: RunStatus::Running,
1157                label: None,
1158                initiator: Initiator::Human {
1159                    camp: "test".to_string(),
1160                },
1161                beholder_status: None,
1162                pinned: false,
1163                origin: origin.map(str::to_string),
1164                host_pid,
1165            })
1166            .await
1167            .unwrap();
1168        id
1169    }
1170
1171    async fn is_lost(store: &Arc<TaskStore>, id: &TaskRunId) -> bool {
1172        matches!(
1173            store.get_run(id).await.unwrap().unwrap().status,
1174            RunStatus::Lost { .. }
1175        )
1176    }
1177
1178    fn adopt_terminal() -> StaleRunPolicy {
1179        StaleRunPolicy::AdoptLiveHosts {
1180            origins: vec!["terminal".to_string()],
1181        }
1182    }
1183
1184    /// The property the whole ticket exists for: attaching to a store must not
1185    /// declare another live process's shell dead.
1186    #[tokio::test]
1187    async fn a_run_owned_by_a_live_host_survives_a_new_driver() {
1188        let dir = tempfile::tempdir().unwrap();
1189        let store = open_store(&dir).await;
1190        // Our own pid is by definition a live process, and is the cheapest
1191        // honest stand-in for "a peer that is still running".
1192        let id = plant_running(&store, Some("terminal"), Some(std::process::id())).await;
1193
1194        let _driver = TaskDriver::with_config(
1195            Arc::clone(&store),
1196            DriverChannels::default(),
1197            adopt_terminal(),
1198        )
1199        .await
1200        .unwrap();
1201
1202        assert!(
1203            !is_lost(&store, &id).await,
1204            "a terminal run whose owner is alive must stay Running — \
1205             tombstoning it is what made a surviving shell read as dead"
1206        );
1207    }
1208
1209    /// The other half: a genuinely abandoned shell must still be tombstoned,
1210    /// or a crashed host leaves permanent zombie tiles.
1211    #[tokio::test]
1212    async fn a_run_whose_host_is_gone_is_still_tombstoned() {
1213        let dir = tempfile::tempdir().unwrap();
1214        let store = open_store(&dir).await;
1215        // Reaped in-test, so the pid is real-but-dead rather than guessed.
1216        let dead_pid = {
1217            let child = std::process::Command::new("true").spawn().unwrap();
1218            let pid = child.id();
1219            let mut child = child;
1220            let _ = child.wait();
1221            pid
1222        };
1223        let id = plant_running(&store, Some("terminal"), Some(dead_pid)).await;
1224
1225        let _driver = TaskDriver::with_config(
1226            Arc::clone(&store),
1227            DriverChannels::default(),
1228            adopt_terminal(),
1229        )
1230        .await
1231        .unwrap();
1232
1233        assert!(
1234            is_lost(&store, &id).await,
1235            "pid {dead_pid} was reaped; its run has no owner left and must be Lost"
1236        );
1237    }
1238
1239    /// The exemption is narrowed by origin, so ordinary jobs keep the old rule
1240    /// even when their owner happens to still be alive — an in-flight `cargo
1241    /// build` whose driver is gone has nobody left to record its exit.
1242    #[tokio::test]
1243    async fn a_non_matching_origin_is_tombstoned_even_with_a_live_host() {
1244        let dir = tempfile::tempdir().unwrap();
1245        let store = open_store(&dir).await;
1246        let job = plant_running(&store, None, Some(std::process::id())).await;
1247        let other = plant_running(&store, Some("gnome"), Some(std::process::id())).await;
1248
1249        let _driver = TaskDriver::with_config(
1250            Arc::clone(&store),
1251            DriverChannels::default(),
1252            adopt_terminal(),
1253        )
1254        .await
1255        .unwrap();
1256
1257        assert!(is_lost(&store, &job).await, "an origin-less job is not exempt");
1258        assert!(
1259            is_lost(&store, &other).await,
1260            "an origin outside the list is not exempt"
1261        );
1262    }
1263
1264    /// A row written before `host_pid` existed reads back `None`. Unknown
1265    /// ownership must fall back to the old behaviour rather than stranding the
1266    /// run `Running` forever.
1267    #[tokio::test]
1268    async fn an_unattributed_run_is_tombstoned() {
1269        let dir = tempfile::tempdir().unwrap();
1270        let store = open_store(&dir).await;
1271        let id = plant_running(&store, Some("terminal"), None).await;
1272
1273        let _driver = TaskDriver::with_config(
1274            Arc::clone(&store),
1275            DriverChannels::default(),
1276            adopt_terminal(),
1277        )
1278        .await
1279        .unwrap();
1280
1281        assert!(is_lost(&store, &id).await);
1282    }
1283
1284    /// `TaskDriver::new` must not have quietly changed behaviour — every
1285    /// existing embedder still gets Lost-on-disappear.
1286    #[tokio::test]
1287    async fn the_default_policy_is_still_lost_on_disappear() {
1288        assert_eq!(StaleRunPolicy::default(), StaleRunPolicy::LostOnDisappear);
1289
1290        let dir = tempfile::tempdir().unwrap();
1291        let store = open_store(&dir).await;
1292        let id = plant_running(&store, Some("terminal"), Some(std::process::id())).await;
1293
1294        let _driver = TaskDriver::new(Arc::clone(&store)).await.unwrap();
1295
1296        assert!(
1297            is_lost(&store, &id).await,
1298            "the default must tombstone regardless of origin or owner liveness"
1299        );
1300    }
1301
1302    /// The owner is recorded by `spawn_run` itself, not by the caller — the
1303    /// policy is worthless if rows arrive unattributed.
1304    #[tokio::test]
1305    async fn spawn_run_stamps_this_process_as_the_owner() {
1306        let dir = tempfile::tempdir().unwrap();
1307        let store = open_store(&dir).await;
1308        let driver = TaskDriver::new(Arc::clone(&store)).await.unwrap();
1309
1310        let id = driver
1311            .spawn_run(
1312                "true",
1313                SpawnOpts {
1314                    cwd: "/tmp".into(),
1315                    origin: Some("terminal".to_string()),
1316                    ..Default::default()
1317                },
1318            )
1319            .await
1320            .unwrap();
1321
1322        let meta = store.get_run(&id).await.unwrap().unwrap();
1323        assert_eq!(meta.host_pid, Some(std::process::id()));
1324    }
1325
1326    #[tokio::test]
1327    async fn new_driver_does_not_touch_completed_runs() {
1328        let dir = tempfile::tempdir().unwrap();
1329        let store = open_store(&dir).await;
1330
1331        let done_id = TaskRunId::new();
1332        store
1333            .insert_run(&TaskRunMeta {
1334                id: done_id.clone(),
1335                command: "true".to_string(),
1336                cwd: "/tmp".into(),
1337                env: vec![],
1338                started_at: unix_now_secs() - 10,
1339                status: RunStatus::Running,
1340                label: None,
1341                initiator: Initiator::Human { camp: "test".to_string() },
1342                beholder_status: None,
1343                pinned: false,
1344                origin: None,
1345                host_pid: None,
1346            })
1347            .await
1348            .unwrap();
1349        store
1350            .update_status(&done_id, &RunStatus::Done { exit_code: 0, ended_at: unix_now_secs() })
1351            .await
1352            .unwrap();
1353
1354        let _driver = TaskDriver::new(Arc::clone(&store)).await.unwrap();
1355
1356        let meta = store.get_run(&done_id).await.unwrap().unwrap();
1357        assert!(
1358            matches!(meta.status, RunStatus::Done { .. }),
1359            "completed run must not be touched"
1360        );
1361    }
1362
1363    // ── PTY spawn + capture ───────────────────────────────────────────────────
1364
1365    #[tokio::test]
1366    async fn spawn_echo_and_read_chunks() {
1367        let dir = tempfile::tempdir().unwrap();
1368        let store = open_store(&dir).await;
1369        let driver = TaskDriver::new(Arc::clone(&store)).await.unwrap();
1370
1371        let id = driver
1372            .spawn_run(
1373                "echo hello_world",
1374                SpawnOpts { cwd: "/tmp".into(), ..Default::default() },
1375            )
1376            .await
1377            .unwrap();
1378
1379        // Wait for the run to complete (poll status up to 5 s).
1380        let deadline = std::time::Instant::now() + Duration::from_secs(5);
1381        loop {
1382            let meta = store.get_run(&id).await.unwrap().unwrap();
1383            if matches!(meta.status, RunStatus::Done { .. } | RunStatus::Lost { .. }) {
1384                break;
1385            }
1386            if std::time::Instant::now() > deadline {
1387                panic!("run did not complete in time, status={:?}", meta.status);
1388            }
1389            tokio::time::sleep(Duration::from_millis(50)).await;
1390        }
1391
1392        // Chunks must contain "hello_world".
1393        let chunks = store
1394            .get_chunks(&id, &ChunkFilter::default())
1395            .await
1396            .unwrap();
1397        let output: Vec<u8> = chunks.into_iter().flat_map(|c| c.bytes).collect();
1398        let text = String::from_utf8_lossy(&output);
1399        assert!(
1400            text.contains("hello_world"),
1401            "expected 'hello_world' in output, got: {text:?}"
1402        );
1403
1404        let meta = store.get_run(&id).await.unwrap().unwrap();
1405        assert!(
1406            matches!(meta.status, RunStatus::Done { exit_code: 0, .. }),
1407            "expected Done(0), got {:?}",
1408            meta.status
1409        );
1410    }
1411
1412    /// Wait for a run to reach a terminal status, or panic.
1413    async fn await_done(store: &TaskStore, id: &TaskRunId) -> TaskRunMeta {
1414        let deadline = std::time::Instant::now() + Duration::from_secs(5);
1415        loop {
1416            let meta = store.get_run(id).await.unwrap().unwrap();
1417            if matches!(meta.status, RunStatus::Done { .. } | RunStatus::Lost { .. }) {
1418                return meta;
1419            }
1420            if std::time::Instant::now() > deadline {
1421                panic!("run did not complete in time, status={:?}", meta.status);
1422            }
1423            tokio::time::sleep(Duration::from_millis(50)).await;
1424        }
1425    }
1426
1427    async fn output_of(store: &TaskStore, id: &TaskRunId) -> String {
1428        let chunks = store.get_chunks(id, &ChunkFilter::default()).await.unwrap();
1429        let bytes: Vec<u8> = chunks.into_iter().flat_map(|c| c.bytes).collect();
1430        String::from_utf8_lossy(&bytes).into_owned()
1431    }
1432
1433    // ── The caller's bytes reach the shell unchanged (R739-S2) ───────────────
1434
1435    /// `AttachResult.argv` is populated on every run, rewrite or not, so
1436    /// `spawn_run` used to join it back into the command line unconditionally.
1437    /// That put every `task.run` command through a whitespace normalization
1438    /// nobody asked for. A multi-line command is the case where that is not
1439    /// cosmetic: the newline the caller wrote becomes a space, and two
1440    /// commands become one nonsense command.
1441    #[cfg(unix)]
1442    #[tokio::test]
1443    async fn a_multi_line_command_is_not_flattened_into_one_line() {
1444        let dir = tempfile::tempdir().unwrap();
1445        let store = open_store(&dir).await;
1446        let driver = Arc::new(TaskDriver::new(Arc::clone(&store)).await.unwrap());
1447
1448        // Flattened to one line this is `echo one echo two`, which prints
1449        // "one echo two" — a different answer, not a failure, which is what
1450        // makes the old behaviour dangerous rather than merely wrong.
1451        let id = driver
1452            .spawn_run(
1453                "echo one\necho two",
1454                SpawnOpts { cwd: "/tmp".into(), ..Default::default() },
1455            )
1456            .await
1457            .unwrap();
1458        await_done(&store, &id).await;
1459
1460        let out = output_of(&store, &id).await;
1461        assert!(out.contains("one"), "got: {out:?}");
1462        assert!(
1463            out.contains("two"),
1464            "the second line must have run as its own command; got: {out:?}"
1465        );
1466        assert!(
1467            !out.contains("one echo two"),
1468            "the newline was flattened into a space; got: {out:?}"
1469        );
1470    }
1471
1472    /// `resolve_argv` strips `bunx`/`npx`/`pnpm` so a beholder's `matches` sees
1473    /// the bare tool. That is a *matching* concern; it must never reach the
1474    /// spawn, or the wrapper the caller needed is gone from the command.
1475    #[cfg(unix)]
1476    #[tokio::test]
1477    async fn a_wrapper_the_caller_wrote_is_not_stripped_from_the_spawned_command() {
1478        let dir = tempfile::tempdir().unwrap();
1479        let store = open_store(&dir).await;
1480        let driver = Arc::new(TaskDriver::new(Arc::clone(&store)).await.unwrap());
1481
1482        // `npx` is almost certainly absent in test environments, and that is
1483        // the point: if the wrapper survived, the shell reports it missing. If
1484        // it were stripped we would be running bare `--version`.
1485        let id = driver
1486            .spawn_run(
1487                "npx r739s2-nonexistent-tool --version",
1488                SpawnOpts { cwd: "/tmp".into(), ..Default::default() },
1489            )
1490            .await
1491            .unwrap();
1492        let meta = await_done(&store, &id).await;
1493        let out = output_of(&store, &id).await;
1494        assert!(
1495            !matches!(meta.status, RunStatus::Done { exit_code: 0, .. }),
1496            "expected a failure, got {:?} with output {out:?}",
1497            meta.status
1498        );
1499        assert!(
1500            !out.contains("--version: "),
1501            "the wrapper was stripped and the shell tried to run the flag; got: {out:?}"
1502        );
1503    }
1504
1505    // ── Direct argv (R652-T6) ────────────────────────────────────────────────
1506
1507    #[tokio::test]
1508    async fn explicit_argv_execs_the_program_directly() {
1509        let dir = tempfile::tempdir().unwrap();
1510        let store = open_store(&dir).await;
1511        let driver = TaskDriver::new(Arc::clone(&store)).await.unwrap();
1512
1513        /* The distinguishing observation: under `sh -c` the child is `sh` and
1514           `$0` is `sh`; exec'd directly it is the program itself. Printing
1515           `$0` is the cheapest way to see which of the two happened. */
1516        let id = driver
1517            .spawn_run(
1518                "unused-because-argv-wins",
1519                SpawnOpts {
1520                    cwd: "/tmp".into(),
1521                    argv: Some(vec![
1522                        "/bin/sh".into(),
1523                        "-c".into(),
1524                        "printf 'argv0=%s\\n' \"$0\"".into(),
1525                        "direct-exec-marker".into(),
1526                    ]),
1527                    ..Default::default()
1528                },
1529            )
1530            .await
1531            .unwrap();
1532
1533        await_done(&store, &id).await;
1534        let text = output_of(&store, &id).await;
1535        assert!(
1536            text.contains("argv0=direct-exec-marker"),
1537            "argv should have been exec'd verbatim, got: {text:?}"
1538        );
1539    }
1540
1541    #[tokio::test]
1542    async fn explicit_argv_still_records_the_requested_command() {
1543        let dir = tempfile::tempdir().unwrap();
1544        let store = open_store(&dir).await;
1545        let driver = TaskDriver::new(Arc::clone(&store)).await.unwrap();
1546
1547        /* A shell tile asks for "$SHELL" and the daemon resolves it to a real
1548           argv. The run must still read back as what was asked for, or the
1549           rail row and the history re-run both show an implementation
1550           detail. */
1551        let id = driver
1552            .spawn_run(
1553                "$SHELL",
1554                SpawnOpts {
1555                    cwd: "/tmp".into(),
1556                    argv: Some(vec!["/bin/sh".into(), "-c".into(), "true".into()]),
1557                    ..Default::default()
1558                },
1559            )
1560            .await
1561            .unwrap();
1562
1563        let meta = await_done(&store, &id).await;
1564        assert_eq!(meta.command, "$SHELL");
1565        assert!(
1566            matches!(meta.status, RunStatus::Done { exit_code: 0, .. }),
1567            "expected Done(0), got {:?}",
1568            meta.status
1569        );
1570    }
1571
1572    #[tokio::test]
1573    async fn empty_argv_falls_back_to_the_shell_path() {
1574        let dir = tempfile::tempdir().unwrap();
1575        let store = open_store(&dir).await;
1576        let driver = TaskDriver::new(Arc::clone(&store)).await.unwrap();
1577
1578        let id = driver
1579            .spawn_run(
1580                "echo empty_argv_fallback",
1581                SpawnOpts { cwd: "/tmp".into(), argv: Some(vec![]), ..Default::default() },
1582            )
1583            .await
1584            .unwrap();
1585
1586        await_done(&store, &id).await;
1587        let text = output_of(&store, &id).await;
1588        assert!(
1589            text.contains("empty_argv_fallback"),
1590            "empty argv must not spawn nothing, got: {text:?}"
1591        );
1592    }
1593
1594    #[tokio::test]
1595    async fn spawn_failing_command_records_nonzero_exit() {
1596        let dir = tempfile::tempdir().unwrap();
1597        let store = open_store(&dir).await;
1598        let driver = TaskDriver::new(Arc::clone(&store)).await.unwrap();
1599
1600        let id = driver
1601            .spawn_run(
1602                "exit 42",
1603                SpawnOpts { cwd: "/tmp".into(), ..Default::default() },
1604            )
1605            .await
1606            .unwrap();
1607
1608        let deadline = std::time::Instant::now() + Duration::from_secs(5);
1609        loop {
1610            let meta = store.get_run(&id).await.unwrap().unwrap();
1611            if !matches!(meta.status, RunStatus::Running | RunStatus::Pending) {
1612                match meta.status {
1613                    RunStatus::Done { exit_code, .. } => {
1614                        assert_ne!(exit_code, 0, "exit 42 should produce a non-zero exit code");
1615                    }
1616                    other => panic!("unexpected status: {other:?}"),
1617                }
1618                break;
1619            }
1620            if std::time::Instant::now() > deadline {
1621                panic!("run did not complete in time");
1622            }
1623            tokio::time::sleep(Duration::from_millis(50)).await;
1624        }
1625    }
1626
1627    // ── Signal handling ───────────────────────────────────────────────────────
1628
1629    #[cfg(unix)]
1630    #[tokio::test]
1631    async fn kill_with_sigterm_transitions_to_killed() {
1632        let dir = tempfile::tempdir().unwrap();
1633        let store = open_store(&dir).await;
1634        let driver = Arc::new(TaskDriver::new(Arc::clone(&store)).await.unwrap());
1635
1636        let id = driver
1637            .spawn_run(
1638                "sleep 60",
1639                SpawnOpts { cwd: "/tmp".into(), ..Default::default() },
1640            )
1641            .await
1642            .unwrap();
1643
1644        // Give the process a moment to start.
1645        tokio::time::sleep(Duration::from_millis(100)).await;
1646
1647        driver.kill_run(&id, Some(SIGTERM)).await.unwrap();
1648
1649        let deadline = std::time::Instant::now() + Duration::from_secs(10);
1650        loop {
1651            let meta = store.get_run(&id).await.unwrap().unwrap();
1652            if matches!(meta.status, RunStatus::Killed { .. } | RunStatus::Lost { .. }) {
1653                assert!(
1654                    matches!(meta.status, RunStatus::Killed { .. }),
1655                    "expected Killed, got {:?}",
1656                    meta.status
1657                );
1658                break;
1659            }
1660            if std::time::Instant::now() > deadline {
1661                panic!("run did not become Killed in time, status={:?}", meta.status);
1662            }
1663            tokio::time::sleep(Duration::from_millis(50)).await;
1664        }
1665    }
1666
1667    #[cfg(unix)]
1668    #[tokio::test]
1669    async fn kill_run_returns_not_found_after_exit() {
1670        let dir = tempfile::tempdir().unwrap();
1671        let store = open_store(&dir).await;
1672        let driver = Arc::new(TaskDriver::new(Arc::clone(&store)).await.unwrap());
1673
1674        let id = driver
1675            .spawn_run(
1676                "echo done",
1677                SpawnOpts { cwd: "/tmp".into(), ..Default::default() },
1678            )
1679            .await
1680            .unwrap();
1681
1682        // Wait for natural exit.
1683        let deadline = std::time::Instant::now() + Duration::from_secs(5);
1684        loop {
1685            let meta = store.get_run(&id).await.unwrap().unwrap();
1686            if !matches!(meta.status, RunStatus::Running | RunStatus::Pending) {
1687                break;
1688            }
1689            if std::time::Instant::now() > deadline {
1690                panic!("run did not complete");
1691            }
1692            tokio::time::sleep(Duration::from_millis(50)).await;
1693        }
1694
1695        // Kill on a completed run should return NotFound.
1696        let result = driver.kill_run(&id, None).await;
1697        assert!(
1698            matches!(result, Err(DriverError::NotFound(_))),
1699            "expected NotFound, got {result:?}"
1700        );
1701    }
1702
1703    // ── Stdin relay ───────────────────────────────────────────────────────────
1704
1705    #[cfg(unix)]
1706    #[tokio::test]
1707    async fn stdin_send_reaches_child() {
1708        let dir = tempfile::tempdir().unwrap();
1709        let store = open_store(&dir).await;
1710        let driver = Arc::new(TaskDriver::new(Arc::clone(&store)).await.unwrap());
1711
1712        // Shell that reads a line from stdin and echoes it back.
1713        let id = driver
1714            .spawn_run(
1715                "read line && echo got_$line",
1716                SpawnOpts {
1717                    cwd: "/tmp".into(),
1718                    stdin_enabled: true,
1719                    ..Default::default()
1720                },
1721            )
1722            .await
1723            .unwrap();
1724
1725        tokio::time::sleep(Duration::from_millis(150)).await;
1726        driver.send_stdin(&id, b"hello\n".to_vec()).await.unwrap();
1727
1728        let deadline = std::time::Instant::now() + Duration::from_secs(5);
1729        loop {
1730            let meta = store.get_run(&id).await.unwrap().unwrap();
1731            if !matches!(meta.status, RunStatus::Running | RunStatus::Pending) {
1732                break;
1733            }
1734            if std::time::Instant::now() > deadline {
1735                panic!("run did not complete after stdin input");
1736            }
1737            tokio::time::sleep(Duration::from_millis(50)).await;
1738        }
1739
1740        let chunks = store.get_chunks(&id, &ChunkFilter::default()).await.unwrap();
1741        let raw: Vec<u8> = chunks.into_iter().flat_map(|c| c.bytes).collect();
1742        let text = String::from_utf8_lossy(&raw);
1743        assert!(
1744            text.contains("got_hello"),
1745            "expected 'got_hello' in output, got: {text:?}"
1746        );
1747    }
1748
1749    /// `resize_run` must change the geometry the *child* sees, not just the
1750    /// master fd — so the assertion reads `stty size` from inside the PTY
1751    /// after the resize rather than inspecting the driver's own state.
1752    #[tokio::test]
1753    async fn resize_run_changes_geometry_the_child_sees() {
1754        let dir = tempfile::tempdir().unwrap();
1755        let store = open_store(&dir).await;
1756        let driver = Arc::new(TaskDriver::new(Arc::clone(&store)).await.unwrap());
1757
1758        // Wait for a line on stdin, then report the geometry as of that moment.
1759        let id = driver
1760            .spawn_run(
1761                "read line && stty size",
1762                SpawnOpts {
1763                    cwd: "/tmp".into(),
1764                    stdin_enabled: true,
1765                    // Spawn at the default 80x24 so the assertion can't pass by
1766                    // accident if the resize is a no-op.
1767                    ..Default::default()
1768                },
1769            )
1770            .await
1771            .unwrap();
1772
1773        tokio::time::sleep(Duration::from_millis(150)).await;
1774        driver.resize_run(&id, 120, 40).await.unwrap();
1775        driver.send_stdin(&id, b"go\n".to_vec()).await.unwrap();
1776
1777        let deadline = std::time::Instant::now() + Duration::from_secs(5);
1778        loop {
1779            let meta = store.get_run(&id).await.unwrap().unwrap();
1780            if !matches!(meta.status, RunStatus::Running | RunStatus::Pending) {
1781                break;
1782            }
1783            if std::time::Instant::now() > deadline {
1784                panic!("run did not complete after stdin input");
1785            }
1786            tokio::time::sleep(Duration::from_millis(50)).await;
1787        }
1788
1789        let chunks = store.get_chunks(&id, &ChunkFilter::default()).await.unwrap();
1790        let raw: Vec<u8> = chunks.into_iter().flat_map(|c| c.bytes).collect();
1791        let text = String::from_utf8_lossy(&raw);
1792        assert!(
1793            text.contains("40 120"),
1794            "expected resized geometry '40 120' in output, got: {text:?}"
1795        );
1796    }
1797
1798    /// A run that is not active on this driver (finished, or never existed) is
1799    /// `NotFound` rather than a panic — same contract as `send_stdin`.
1800    #[tokio::test]
1801    async fn resize_run_returns_not_found_after_exit() {
1802        let dir = tempfile::tempdir().unwrap();
1803        let store = open_store(&dir).await;
1804        let driver = Arc::new(TaskDriver::new(Arc::clone(&store)).await.unwrap());
1805
1806        let id = driver
1807            .spawn_run("true", SpawnOpts { cwd: "/tmp".into(), ..Default::default() })
1808            .await
1809            .unwrap();
1810
1811        let deadline = std::time::Instant::now() + Duration::from_secs(5);
1812        loop {
1813            let meta = store.get_run(&id).await.unwrap().unwrap();
1814            if !matches!(meta.status, RunStatus::Running | RunStatus::Pending) {
1815                break;
1816            }
1817            if std::time::Instant::now() > deadline {
1818                panic!("run did not exit");
1819            }
1820            tokio::time::sleep(Duration::from_millis(50)).await;
1821        }
1822
1823        assert!(matches!(
1824            driver.resize_run(&id, 100, 30).await,
1825            Err(DriverError::NotFound(_))
1826        ));
1827    }
1828
1829    // ── Tier-2 side-channel log fd ────────────────────────────────────────────
1830
1831    /// Verify that a child writing a JSON-line to `YAH_LOG_PIPE` (via
1832    /// `printf ... >> $YAH_LOG_PIPE`) produces a shim event with the correct
1833    /// fields in the store.
1834    ///
1835    /// The child opens the FIFO path for writing — no fd inheritance needed.
1836    #[cfg(unix)]
1837    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
1838    async fn log_pipe_events_land_in_store() {
1839        use crate::store::EventFilter;
1840
1841        let dir = tempfile::tempdir().unwrap();
1842        let store = open_store(&dir).await;
1843        let driver = Arc::new(TaskDriver::new(Arc::clone(&store)).await.unwrap());
1844
1845        // The shell writes one JSON-line to the FIFO by redirecting printf
1846        // output to the path stored in YAH_LOG_PIPE.
1847        let cmd = r#"printf '{"level":"warn","target":"test.shim","msg":"hello-from-pipe","fields":{"x":42},"_lib":"test-shim","_lib_ver":"0.1.0"}\n' >> "$YAH_LOG_PIPE""#;
1848
1849        let id = driver
1850            .spawn_run(cmd, SpawnOpts { cwd: "/tmp".into(), ..Default::default() })
1851            .await
1852            .unwrap();
1853
1854        // Wait for run completion. Deadline is generous because parallel-test
1855        // load + the rt.block_on hops from the reader/log threads can slow
1856        // child-process scheduling.
1857        let deadline = std::time::Instant::now() + Duration::from_secs(20);
1858        loop {
1859            let meta = store.get_run(&id).await.unwrap().unwrap();
1860            if matches!(meta.status, RunStatus::Done { .. } | RunStatus::Lost { .. }) {
1861                break;
1862            }
1863            if std::time::Instant::now() > deadline {
1864                panic!("run did not complete in time");
1865            }
1866            tokio::time::sleep(Duration::from_millis(50)).await;
1867        }
1868
1869        // The log receiver thread drains after the lifecycle task drops the
1870        // write-end FdCloser; give it a brief moment.
1871        tokio::time::sleep(Duration::from_millis(500)).await;
1872
1873        let events = store.query_events(&id, &EventFilter::default()).await.unwrap();
1874        assert!(
1875            !events.is_empty(),
1876            "expected at least one shim event, got none"
1877        );
1878        let ev = events.iter().find(|e| e.target == "test.shim");
1879        let ev = ev.expect("event with target 'test.shim' not found");
1880        assert_eq!(ev.msg, "hello-from-pipe");
1881        assert_eq!(ev.level, crate::types::Level::Warn);
1882        assert!(
1883            matches!(&ev.source, crate::types::EventSource::Shim { lib, .. } if lib == "test-shim"),
1884            "unexpected source: {:?}",
1885            ev.source
1886        );
1887        assert_eq!(ev.fields.get("x"), Some(&serde_json::json!(42)));
1888    }
1889
1890    /// When `log_fd_enabled` is false, neither `YAH_TASK_RUN` nor
1891    /// `YAH_LOG_PIPE` are exported, and no shim events are written.
1892    #[cfg(unix)]
1893    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
1894    async fn log_pipe_disabled_produces_no_events() {
1895        use crate::store::EventFilter;
1896
1897        let dir = tempfile::tempdir().unwrap();
1898        let store = open_store(&dir).await;
1899        let driver = Arc::new(TaskDriver::new(Arc::clone(&store)).await.unwrap());
1900
1901        // Try to write to YAH_LOG_PIPE; the conditional guards against
1902        // the variable being absent, so the command always exits 0.
1903        let cmd = r#"[ -n "$YAH_LOG_PIPE" ] && printf '{"level":"info","target":"t","msg":"m","fields":{}}\n' >> "$YAH_LOG_PIPE" || true"#;
1904
1905        let id = driver
1906            .spawn_run(
1907                cmd,
1908                SpawnOpts { cwd: "/tmp".into(), log_fd_enabled: false, ..Default::default() },
1909            )
1910            .await
1911            .unwrap();
1912
1913        let deadline = std::time::Instant::now() + Duration::from_secs(5);
1914        loop {
1915            let meta = store.get_run(&id).await.unwrap().unwrap();
1916            if matches!(meta.status, RunStatus::Done { .. } | RunStatus::Lost { .. }) {
1917                break;
1918            }
1919            if std::time::Instant::now() > deadline {
1920                panic!("run did not complete");
1921            }
1922            tokio::time::sleep(Duration::from_millis(50)).await;
1923        }
1924
1925        tokio::time::sleep(Duration::from_millis(100)).await;
1926
1927        let events = store.query_events(&id, &EventFilter::default()).await.unwrap();
1928        assert!(
1929            events.is_empty(),
1930            "expected no shim events when log_fd_enabled=false, got {}",
1931            events.len()
1932        );
1933    }
1934}