shep-daemon 0.2.0

The supervision engine behind shep: process spawning, restart policy, the log plane, watch and cron restarts, and the RPC server
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
//! The `Prober` seam and the liveness probe loop (spec §7).
//!
//! `spawn_liveness_task` runs one sheep's readiness/liveness probe on
//! [`ProbeConfig::interval`](shep_core::config::ProbeConfig::interval),
//! reporting through a shared channel once
//! [`ProbeConfig::failure_threshold`](shep_core::config::ProbeConfig::failure_threshold)
//! *consecutive* probes have failed, then ending: a sheep already declared
//! unhealthy is about to be restarted, and the loop for its replacement —
//! armed against a new pid — is a new one.
//!
//! # Which readiness source wins
//!
//! An app can name two, and `ready::ReadinessSource::of` picks exactly one:
//!
//! 1. **`wait_ready = true`** — the shepherd channel's `{"kind":"ready"}`.
//!    Chosen whenever it is set, even alongside a `readiness_probe`: the
//!    channel is the app telling us directly, and a probe is an outside guess
//!    at the same fact.
//! 2. **`readiness_probe`** — the first probe that passes.
//! 3. **Neither** — a plain `start` takes the sheep `online` at spawn. (The
//!    `Heuristic` source, which waits out `listen_timeout`, is reload's; it
//!    is fully implemented here but `start` never selects it.)
//!
//! `failure_threshold` is deliberately not consulted while gating readiness.
//! It is a liveness concept, and giving up after N failed readiness probes
//! would nest a second, smaller deadline inside `listen_timeout`.
//!
//! # A readiness timeout is not a liveness failure
//!
//! They look alike and mean opposite things:
//!
//! | | readiness timeout | liveness failure |
//! |---|---|---|
//! | When | `listen_timeout` elapses before the app signals or a probe passes | `failure_threshold` *consecutive* probes fail on a sheep already `online` |
//! | Verdict | the app is **slow** | the app is **wedged** |
//! | Result | the sheep goes `online` anyway | the sheep is restarted |
//! | Budget | untouched | **reset** — a liveness restart is a restart command, and every one of those resets it |
//!
//! The readiness timeout does log a `warn!` before going online, and that
//! record is the only thing telling the two apart: a sheep that came up
//! because its deadline elapsed reaches the same `online` status, and emits
//! the same bus event, as one that answered. The binary renders it into
//! `$SHEP_HOME/logs/shepd.err.log` at the default `warn` level.
//!
//! Treating a slow start as a failure produces exactly the restart loop
//! `max_restarts` exists to contain, out of an app that is merely slow, so
//! readiness gives up *upward*. Liveness has already had the app answer
//! successfully at least once, so a run of failures is evidence rather than
//! impatience.
//!
//! # Caveats
//!
//! - **No TLS.** `os::OsProber`'s HTTP client is hand-rolled and speaks
//!   cleartext only. `https://` targets are refused in `shep-core` at config
//!   time rather than failing every poll, because a probe that always fails
//!   is indistinguishable from an app that is down (decision D1).
//! - **No redirects.** A `3xx` is a non-2xx and therefore a failed probe.
//!   Point the probe at the endpoint, not at something that forwards to it.
//! - **`timeout` is the implementation's promise, not this loop's.** See
//!   [`Prober`]'s own design note: a `probe` future that never resolves
//!   stalls liveness detection for that sheep forever.
//! - **Granularity is `interval`, floored at `MIN_PROBE_INTERVAL`.** Worst
//!   case, a wedged app is noticed `failure_threshold × interval` after it
//!   wedges.
//!
//! ## Reference
//!
//! - [`Prober`], [`ProbeFailure`]
//! - `LivenessFailure`, `spawn_liveness_task`, `MIN_PROBE_INTERVAL`
//! - `os::OsProber` — the real HTTP/TCP/exec implementation
//! - `ready::ReadinessSource`, `ready::Readiness`, `ready::await_ready`
//!   — the `starting → online` gate (spec §7)
//! - [`shep_core::config::ProbeTarget`] — where a target is parsed and an
//!   `https://` one is refused

pub(crate) mod os;
pub(crate) mod ready;

use core::future::Future;
use core::pin::Pin;
use core::time::Duration;

use std::sync::Arc;

use tokio::sync::mpsc;

use shep_core::config::{ProbeConfig, ProbeTarget};

/// Why a probe did not pass.
///
/// Growth is expected: each new probe transport brings its own failure modes
/// (IR-20).
#[non_exhaustive]
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ProbeFailure {
    /// The probe did not finish inside `ProbeConfig::timeout`.
    Timeout,
    /// The transport failed before a verdict was possible — connection
    /// refused, DNS failure, the command could not be spawned. Carries the
    /// rendered reason.
    Transport(String),
    /// The probe completed and the answer was negative: a non-2xx status, or a
    /// non-zero exit. Carries the status or exit code.
    Rejected(String),
}

// A boxed future rather than RPITIT, because `Arc<dyn Prober>` is how the
// engine holds this and RPITIT is not dyn-compatible. One allocation per probe
// — once per `interval`, default 10s — against three extra generic parameters
// threaded through the actor and every fixture.
/// Runs one probe against one target.
///
/// # Design note: `timeout` enforcement is the implementation's job
///
/// `spawn_liveness_task` awaits [`Self::probe`] directly; it does not
/// additionally wrap the call in its own `tokio::time::timeout`. That means
/// a `Prober` whose `probe` future never resolves — hangs rather than
/// erroring — stalls the liveness loop forever: no further probes, no
/// report, ever. This is an accepted design risk, not a defect (bounding
/// every code path by `timeout` needs implementation-specific knowledge,
/// e.g. a connect timeout versus a read timeout, that this seam has no
/// business dictating) — but any implementor (`os::OsProber` chief among
/// them) must itself guarantee `probe` resolves within `timeout` on
/// every path, or a hung sheep's liveness detection silently stops working.
///
/// Public, with [`ProbeFailure`], because `tests/external_impls.rs` implements
/// it from outside this crate — the standing proof that the seam is a seam.
pub trait Prober: Send + Sync + 'static {
    /// Probes `target`, giving up after `timeout`.
    ///
    /// # Errors
    ///
    /// - [`ProbeFailure::Timeout`] — the probe did not finish inside
    ///   `timeout`.
    /// - [`ProbeFailure::Transport`] — the transport failed before a verdict
    ///   was possible (connection refused, DNS failure, the command could
    ///   not be spawned).
    /// - [`ProbeFailure::Rejected`] — the probe completed and the answer was
    ///   negative (a non-2xx status, or a non-zero exit).
    fn probe<'a>(
        &'a self,
        target: &'a ProbeTarget,
        timeout: Duration,
    ) -> Pin<Box<dyn Future<Output = Result<(), ProbeFailure>> + Send + 'a>>;
}

/// Floor `spawn_liveness_task` enforces on `interval`, regardless of caller.
///
/// `shep-core`'s `normalize` already rejects any `interval` under this same
/// value in a Flockfile (`NormalizeError::IntervalBelowMinimum`) — but that
/// guard lives behind boot wiring this module does not own (the same reason
/// `cron::MIN_MAX_SLEEP` keeps its own floor even though `shep-core`
/// separately rejects a too-small `max_cron_sleep`), so it protects only the
/// call site that reaches it, once one exists. Without a floor here too, any
/// caller — today's tests, or a boot path added later that forgets to route
/// through the validated config — could hand this loop a `Duration::ZERO`
/// interval and turn it into a hot spin: measured live at roughly 380 probes
/// per second, which for `ProbeKind::Exec` is that many process spawns per
/// second, per sheep, forever. The value is the same second `shep-core`'s own
/// `MIN_PROBE_INTERVAL` rejects under, so nothing normalization accepts is
/// ever clamped here; it is declared twice because a crate that must not
/// depend on this one owns the other copy, exactly as `MIN_CRON_SLEEP` and
/// `cron::MIN_MAX_SLEEP` are.
const MIN_PROBE_INTERVAL: Duration = Duration::from_millis(1_000);

/// A sheep whose liveness probe hit `failure_threshold`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) struct LivenessFailure {
    /// The sheep's id.
    pub id: u32,
    /// The pid this loop was armed against, for the same reason
    /// [`LimitBreach::root_pid`](crate::limits::LimitBreach) carries one.
    pub pid: u32,
}

/// Runs a sheep's liveness probe until the returned handle is aborted.
///
/// Reports through `failures` once `failure_threshold` consecutive probes have
/// failed, then ends: a sheep that has been declared unhealthy is about to be
/// restarted, and the loop for its replacement is a new one.
///
/// Must be called from within a Tokio runtime context: it spawns the probing
/// task immediately, the same way `spawn_supervisor`, `spawn_cron_worker` and
/// `PollingEnforcer::start` already document for themselves.
pub(crate) fn spawn_liveness_task(
    id: u32,
    pid: u32,
    config: ProbeConfig,
    target: ProbeTarget,
    prober: Arc<dyn Prober>,
    failures: mpsc::Sender<LivenessFailure>,
) -> tokio::task::JoinHandle<()> {
    let interval = config.interval.as_duration().max(MIN_PROBE_INTERVAL);
    let timeout = config.timeout.as_duration();
    let threshold = config.failure_threshold;
    tokio::spawn(async move {
        let mut consecutive_failures: u32 = 0;
        loop {
            // Interval is measured from the PREVIOUS probe's completion, not
            // from this loop's start: sleeping first, then probing, gives a
            // struggling app a full `interval` of quiet between probes even
            // when the probe itself runs long. A `tokio::time::interval`
            // grid ticking independently of how long the probe takes would
            // instead fire every overdue tick back-to-back once a probe
            // outlasts its own period — its default `MissedTickBehavior::
            // Burst` — which is exactly the shape that turns a slow service
            // into a dead one.
            tokio::time::sleep(interval).await;

            match prober.probe(&target, timeout).await {
                Ok(()) => consecutive_failures = 0,
                Err(_failure) => {
                    consecutive_failures += 1;
                    if consecutive_failures >= threshold {
                        // The receiver may already be gone (the reporting
                        // task shut down alongside the engine);
                        // this loop's job ends either way, so the send
                        // result is intentionally discarded rather than
                        // branched on.
                        let _ = failures.send(LivenessFailure { id, pid }).await;
                        return;
                    }
                }
            }
        }
    })
}

#[cfg(test)]
mod tests {
    use core::time::Duration;

    use std::sync::Arc;

    use tokio::sync::mpsc;

    use super::*;
    use crate::testing::{ScriptedProber, probe_config};
    use shep_core::config::ProbeKind;

    /// Generous bound on how long a test may wait for a liveness failure on
    /// the (paused) tokio clock. Costs no real wall-clock time: the paused
    /// runtime auto-advances to this deadline only if nothing else becomes
    /// ready first.
    const FAILURE_WAIT: Duration = Duration::from_secs(120);

    /// The `ProbeTarget` every test below arms against — its contents are
    /// never read: `ScriptedProber::probe` ignores both of its arguments.
    fn target() -> ProbeTarget {
        ProbeTarget::Tcp {
            host: "localhost".to_string(),
            port: 5432,
        }
    }

    /// `config_with_threshold(3)`'s baseline, with `failure_threshold`
    /// overwritten — `probe_config` fixes every other field at
    /// fixture-friendly (production-default) values.
    fn config_with_threshold(threshold: u32) -> ProbeConfig {
        ProbeConfig {
            failure_threshold: threshold,
            ..probe_config(ProbeKind::Tcp, "localhost:5432")
        }
    }

    /// `n` full probe intervals, plus a hair — long enough to be sure `n`
    /// intervals have elapsed and been processed, short enough to stay
    /// strictly before interval `n + 1`. Mirrors `limits::tests::ticks`, but
    /// takes `interval` as a parameter: this loop's cadence is per-config,
    /// not a shared crate-wide constant.
    fn intervals(interval: Duration, n: u32) -> Duration {
        interval * n + Duration::from_millis(1)
    }

    /// Spawns a liveness task and yields once before returning.
    ///
    /// Mirrors `cron::tests::spawn_worker_and_settle` and
    /// `limits::tests::start_and_settle`: a task spawned right before the
    /// clock is driven forward would take its very first `sleep` reading
    /// after that, missing the interval the drive was meant to cross.
    /// Yielding once first lets the loop commit to its initial sleep while
    /// the clock still reads close to "now".
    async fn spawn_and_settle(
        id: u32,
        pid: u32,
        config: ProbeConfig,
        prober: Arc<dyn Prober>,
        failures: mpsc::Sender<LivenessFailure>,
    ) -> tokio::task::JoinHandle<()> {
        let handle = spawn_liveness_task(id, pid, config, target(), prober, failures);
        tokio::task::yield_now().await;
        handle
    }

    async fn expect_failure(rx: &mut mpsc::Receiver<LivenessFailure>, id: u32) -> LivenessFailure {
        match tokio::time::timeout(FAILURE_WAIT, rx.recv()).await {
            Ok(Some(failure)) => failure,
            Ok(None) => panic!("failures channel closed before a failure for id {id} arrived"),
            Err(_) => panic!("timed out waiting for a liveness failure for id {id}"),
        }
    }

    /// Waits up to `window` for a liveness failure, panicking if one
    /// arrives.
    ///
    /// A bounded `timeout` + `recv`, not a bare `try_recv` (Global
    /// Constraints rule 11): right after the clock is driven forward, a
    /// message already due has not necessarily reached this receiver's
    /// queue yet, so a bare `try_recv` reads `Err(Empty)` regardless of
    /// whether the loop under test is correct — it cannot fail, so it
    /// guards nothing.
    async fn assert_no_failure_within(rx: &mut mpsc::Receiver<LivenessFailure>, window: Duration) {
        match tokio::time::timeout(window, rx.recv()).await {
            Err(_) => {} // window elapsed with nothing arriving — expected
            Ok(Some(failure)) => panic!("unexpected liveness failure observed: {failure:?}"),
            Ok(None) => panic!("failures channel disconnected while checking for no failure"),
        }
    }

    /// Dyn-compatibility smoke test (IR-10): fails to compile the moment
    /// somebody adds a generic (non-dyn-safe) method to `Prober`.
    #[test]
    fn prober_is_dyn_compatible() {
        let _: &dyn Prober = &ScriptedProber::new(vec![]);
    }

    // fails if the threshold check is off by one (`>` instead of `>=`), and
    // fails if the loop probes on a `tokio::time::interval` grid instead of
    // sleeping the full `interval` between one probe's completion and the
    // next — a grid loop fires its first probe at t=0 and its third at
    // 2×interval, not 3×interval.
    #[tokio::test(start_paused = true)]
    async fn three_consecutive_failures_report_at_exactly_three_intervals() {
        let config = config_with_threshold(3);
        let interval = config.interval.as_duration();
        let timeout = config.timeout.as_duration();
        let prober = Arc::new(ScriptedProber::new(vec![
            Err(ProbeFailure::Timeout),
            Err(ProbeFailure::Timeout),
            Err(ProbeFailure::Timeout),
        ]));
        let (tx, mut rx) = mpsc::channel(8);
        let start = tokio::time::Instant::now();
        let _handle =
            spawn_and_settle(1, 100, config, Arc::clone(&prober) as Arc<dyn Prober>, tx).await;

        let failure = expect_failure(&mut rx, 1).await;

        assert_eq!(failure, LivenessFailure { id: 1, pid: 100 });
        assert_eq!(
            tokio::time::Instant::now() - start,
            interval * 3,
            "liveness failure should be reported at exactly the third probe, not before or after"
        );
        // fails if `spawn_liveness_task` wires the wrong argument into
        // `prober.probe(&target, ..)` — e.g. `interval` where `timeout`
        // belongs, or `Duration::ZERO` — since nothing else exercises this
        // parameter: `ScriptedProber` ignores it for every other purpose,
        // and `OsProber` is tested standalone, never through this loop.
        assert_eq!(
            prober.last_timeout(),
            timeout,
            "the loop must pass config.timeout, not some other duration, to Prober::probe"
        );
    }

    // No separate "a pass resets the counter" test: that claim is a strict
    // corollary of the test below's, which pins the failure to *exactly*
    // interval 6.
    //
    // fails if the counter resets on a pass but then double-counts
    // afterward, or if a reset counter never re-arms and the loop stops
    // checking the threshold at all
    #[tokio::test(start_paused = true)]
    async fn counter_re_accumulates_after_a_reset_and_trips_on_the_sixth_probe() {
        let config = config_with_threshold(3);
        let interval = config.interval.as_duration();
        let prober = Arc::new(ScriptedProber::new(vec![
            Err(ProbeFailure::Timeout),
            Err(ProbeFailure::Timeout),
            Ok(()),
            Err(ProbeFailure::Timeout),
            Err(ProbeFailure::Timeout),
            Err(ProbeFailure::Timeout),
        ]));
        let (tx, mut rx) = mpsc::channel(8);
        let start = tokio::time::Instant::now();
        let _handle =
            spawn_and_settle(3, 300, config, Arc::clone(&prober) as Arc<dyn Prober>, tx).await;

        let failure = expect_failure(&mut rx, 3).await;

        assert_eq!(failure, LivenessFailure { id: 3, pid: 300 });
        assert_eq!(
            tokio::time::Instant::now() - start,
            interval * 6,
            "the sixth probe (three failures after the reset) should be the one that trips"
        );
    }

    // fails if the comparison is `>` where `>=` belongs: with a threshold of
    // 1, `>` would need a SECOND failure to trip. `ScriptedProber` repeats
    // its last scripted outcome forever, so a bare "did a failure ever
    // arrive" assertion would not catch this — the second failure the `>`
    // mutation waits for is just as available as the first. Pinning the
    // `Instant` at exactly one interval is what makes the difference
    // observable.
    #[tokio::test(start_paused = true)]
    async fn threshold_of_one_reports_after_a_single_failure() {
        let config = config_with_threshold(1);
        let interval = config.interval.as_duration();
        let prober = Arc::new(ScriptedProber::new(vec![Err(ProbeFailure::Timeout)]));
        let (tx, mut rx) = mpsc::channel(8);
        let start = tokio::time::Instant::now();
        let _handle =
            spawn_and_settle(4, 400, config, Arc::clone(&prober) as Arc<dyn Prober>, tx).await;

        let failure = expect_failure(&mut rx, 4).await;

        assert_eq!(failure, LivenessFailure { id: 4, pid: 400 });
        assert_eq!(
            tokio::time::Instant::now() - start,
            interval,
            "threshold of one must report after exactly one probe, not two"
        );
    }

    // fails if the loop keeps probing a sheep it has already declared dead
    // instead of ending once it has reported
    #[tokio::test(start_paused = true)]
    async fn no_further_probing_after_a_failure_is_reported() {
        let config = config_with_threshold(1);
        let interval = config.interval.as_duration();
        let prober = Arc::new(ScriptedProber::new(vec![Err(ProbeFailure::Timeout)]));
        let (tx, mut rx) = mpsc::channel(8);
        // A held clone, not the moved original: `failures` is documented as
        // a shared sender cloned once per arming, and this loop DOES report
        // and end partway through this test, so without a spare clone kept
        // alive here, the channel would close the instant the loop's own
        // sender drops, and the `assert_no_failure_within` below would
        // observe `Ok(None)` (disconnected) instead of a timeout — "channel
        // closed" and "loop correctly silent" must stay distinguishable,
        // and only a live spare sender keeps them so.
        let _handle = spawn_and_settle(
            5,
            500,
            config,
            Arc::clone(&prober) as Arc<dyn Prober>,
            tx.clone(),
        )
        .await;

        expect_failure(&mut rx, 5).await;
        let calls_after_report = prober.calls();

        // A bounded `timeout` clearing the final deadline by a hair, per
        // Global Constraints rule 11 — an explicit `advance` would resolve
        // only the one sleep already pending and could not observe whether
        // the loop issued (and this counter recorded) a probe it should not
        // have.
        assert_no_failure_within(&mut rx, intervals(interval, 3)).await;
        assert_eq!(
            prober.calls(),
            calls_after_report,
            "prober must not be called again after the loop has reported and ended"
        );
    }

    // fails if the loop probes on a `tokio::time::interval` grid instead of
    // sleeping the full `interval` after each probe resolves: a probe that
    // takes 2×interval to resolve would, on that grid, fire every overdue
    // tick back-to-back (`MissedTickBehavior::Burst`) and reach 7 calls in
    // this span, not 4
    #[tokio::test(start_paused = true)]
    async fn a_slow_probe_still_paces_by_interval_after_completion() {
        let config = config_with_threshold(3); // never reached: every probe passes
        let interval = config.interval.as_duration();
        let prober = Arc::new(ScriptedProber::new(vec![Ok(())]).with_delay(interval * 2));
        let (tx, mut rx) = mpsc::channel(8);
        let _handle =
            spawn_and_settle(6, 600, config, Arc::clone(&prober) as Arc<dyn Prober>, tx).await;

        // Cross the span with a bounded timeout that clears the twelfth
        // interval's deadline by a hair, per Global Constraints rule 11 —
        // not `12 × advance(interval)`, which would resolve only the one
        // sleep already pending on each call and undercount every probe
        // this loop actually issued in between.
        assert_no_failure_within(&mut rx, intervals(interval, 12)).await;
        assert_eq!(prober.calls(), 4);
    }

    // fails if the task outlives its sheep: probing continues even after
    // the handle is aborted
    #[tokio::test(start_paused = true)]
    async fn aborting_the_handle_stops_probing() {
        let config = config_with_threshold(1000); // high enough it never trips
        let interval = config.interval.as_duration();
        let prober = Arc::new(ScriptedProber::new(vec![Err(ProbeFailure::Timeout)]));
        let (tx, _rx) = mpsc::channel(8);
        let handle =
            spawn_and_settle(7, 700, config, Arc::clone(&prober) as Arc<dyn Prober>, tx).await;

        tokio::time::sleep(intervals(interval, 2)).await;
        let calls_before_abort = prober.calls();
        assert!(
            calls_before_abort > 0,
            "expected at least one probe before abort"
        );

        handle.abort();

        tokio::time::sleep(intervals(interval, 3)).await;
        assert_eq!(
            prober.calls(),
            calls_before_abort,
            "prober must not be called again once the handle has been aborted"
        );
    }

    // Boundary sweep (IR-40): a `timeout` larger than the `interval`. This
    // subsystem's other two boundaries are already pinned above —
    // `failure_threshold = 1` by `threshold_of_one_reports_after_a_single_failure`,
    // and a zero `interval` by the case below, which also records the
    // decision taken there: a zero interval is FLOORED here and separately
    // REJECTED by `shep-core`'s `normalize`
    // (`NormalizeError::IntervalBelowMinimum`), never trusted as written.
    //
    // fails if the loop clamps `timeout` down to `interval` before handing it
    // to `Prober::probe` — the shape that quietly turns "give this probe five
    // seconds" into "give it two" for any app whose probe is slower than its
    // own polling period. And fails if a probe outlasting its interval makes
    // the loop fire the next one early: on a `tokio::time::interval` grid a
    // 5s probe with a 2s period leaves ticks overdue and bursts them
    // back-to-back, reaching the threshold at 10s rather than the 21s a
    // sleep-after-completion loop takes.
    #[tokio::test(start_paused = true)]
    async fn a_timeout_longer_than_the_interval_is_honoured_and_paces_the_next_probe() {
        let config = ProbeConfig {
            interval: shep_core::values::UpDuration::from_millis(2_000),
            timeout: shep_core::values::UpDuration::from_millis(5_000),
            ..config_with_threshold(3)
        };
        let interval = config.interval.as_duration();
        let timeout = config.timeout.as_duration();
        // A probe that takes its whole timeout to answer is what makes the
        // ordering visible: one cycle then costs `interval + timeout`, a
        // number neither duration produces on its own.
        let prober =
            Arc::new(ScriptedProber::new(vec![Err(ProbeFailure::Timeout)]).with_delay(timeout));
        let (tx, mut rx) = mpsc::channel(8);
        let start = tokio::time::Instant::now();
        let _handle =
            spawn_and_settle(9, 900, config, Arc::clone(&prober) as Arc<dyn Prober>, tx).await;

        let failure = expect_failure(&mut rx, 9).await;

        assert_eq!(failure, LivenessFailure { id: 9, pid: 900 });
        assert_eq!(
            tokio::time::Instant::now() - start,
            (interval + timeout) * 3,
            "three probes that each sleep `interval` and then run for `timeout` must report \
             at 3 x (interval + timeout)"
        );
        assert_eq!(
            prober.last_timeout(),
            timeout,
            "the loop must pass config.timeout unclamped, even where it exceeds interval"
        );
        assert_eq!(prober.calls(), 3);
    }

    // fails if a zero (or otherwise sub-floor) `interval` is trusted instead
    // of clamped to `MIN_PROBE_INTERVAL`: an unclamped `Duration::ZERO`
    // would spin the loop as fast as the runtime allows (see
    // `MIN_PROBE_INTERVAL`'s own doc for what that costs).
    // `shep-core::normalize` rejects an explicit `interval = "0"` in a
    // Flockfile, but this test
    // constructs a `ProbeConfig` directly (as a caller that skipped
    // normalization, or a future boot path with a bug, would) to prove this
    // loop does not simply trust that every caller validated first.
    //
    // Confirmed live (mutation testing, not just written down): with the
    // clamp removed, this test does not fail — it hangs. A `Duration::ZERO`
    // sleep resolves the instant it is polled, on a paused clock exactly as
    // on a real one, so the loop never actually parks on a pending timer for
    // the runtime's auto-advance to fast-forward past; it just spins,
    // consuming real CPU forever. The backstop for that mutation is the CI
    // job's own timeout, the same shape `cron::tests::
    // exhausted_pattern_ends_the_task_without_restarting` already documents
    // for its own busy-spin mutation.
    #[tokio::test(start_paused = true)]
    async fn a_zero_interval_is_clamped_instead_of_hot_spinning() {
        let config = ProbeConfig {
            interval: shep_core::values::UpDuration::from_millis(0),
            ..config_with_threshold(1000) // high enough it never trips
        };
        let prober = Arc::new(ScriptedProber::new(vec![Ok(())]));
        let (tx, _rx) = mpsc::channel(8);
        let _handle =
            spawn_and_settle(8, 800, config, Arc::clone(&prober) as Arc<dyn Prober>, tx).await;

        // Cross exactly 5 floored intervals; a hot-spinning loop would rack
        // up far more than 5 calls in this same span.
        tokio::time::sleep(intervals(MIN_PROBE_INTERVAL, 5)).await;
        assert_eq!(
            prober.calls(),
            5,
            "interval must be clamped to MIN_PROBE_INTERVAL, not trusted as-is"
        );
    }
}