Skip to main content

htl_core/
cache.rs

1//! An on-disk cache of what checking one module reported (issues #3, #19).
2//!
3//! Building a checker costs about 13.5 ms and type-checking a few thousand lines of Teal
4//! costs about a second (`crates/htl-core/benches/check.rs`). A run that changed nothing
5//! should pay neither, and a run that changed one module should pay for that module and
6//! what depends on it rather than for the project.
7//!
8//! # One entry per module
9//!
10//! #3 shipped this as one entry per invocation, because at the time it looked as though
11//! skipping individual modules would change what the rest of them saw. Two things settled
12//! that: #21 made a file's result independent of its position in the walk, and measurement
13//! showed the shared module store to be an optimisation rather than a precondition — a
14//! dependency being checked first does not change what its requirer reports. So an entry
15//! is now per module, and a run is the sum of them.
16//!
17//! # What makes an entry
18//!
19//! The **key** is the module as this invocation names it — the path spelling the checker
20//! will put into the diagnostics, plus the lint selection and the working directory.
21//! `--strict` and `--format` are deliberately absent: they change how a run is summarized
22//! and what it exits with, not what any module reports, so runs that differ only in those
23//! share their modules' entries. A level is in the key because it is written in the same spec as
24//! which rules run — moving one rule between `warn` and `deny` changes no diagnostic, and
25//! re-checks anyway.
26//!
27//! The **inputs** are the module and its require closure, by content hash: every file the
28//! check read through `require`, however many requires away ([`CheckInfo::deps`]). The
29//! closure and not the direct requires, because a check reads further than the module's
30//! own `require` calls. A `global` reaches a file through the whole chain (#370), and a
31//! record a required module re-exports (`type Std = h.Std`) is the record of the file it
32//! came from: `top.tl` requiring `mid.tl` requiring `host.d.tl` is typed by `host.d.tl`
33//! without ever naming it. An entry that hashed only `mid.tl` replayed `top.tl`'s old
34//! answer over an edit to `host.d.tl` — no error where the edit put two, and a
35//! `global-redeclaration` lint from the line numbers `top.tl` had seen before the edit
36//! beside the ones `mid.tl` saw after (#400). The cost is that an edit to a leaf re-checks
37//! everything above it, which is the right answer: the leaf's types reach all of it.
38//! Hashing is microseconds and memoised per run, so a project whose every module sits on
39//! one `.d.tl` hashes that file once.
40//!
41//! The **probes** say what each name the closure required resolves to — the module's own
42//! requires, asked from the module, and every member's, asked from that member
43//! ([`CheckInfo::closure_requires`]). A new `.tl` appearing somewhere changes that while
44//! every recorded hash still matches, and nothing else would catch it. This is the hole
45//! ccache documents in its direct mode, and it is as deep as the closure: a file appearing
46//! under a name `mid.tl` requires changes what `top.tl` sees exactly as an edit does. With
47//! the project's model ([`Cache::with_answers`]) a probe is the model's answer for every
48//! such question, each asked from the file that asked it: one hash, and the questions it
49//! hashes are the ones that matter. Without one (a file outside any project) a probe is a
50//! directory the search path consults — the module's own, and each member's that asked
51//! something — by the set of those names it offers.
52//!
53//! # No mtimes anywhere
54//!
55//! Content hashes only. Timestamp-based invalidation is where this class of tool
56//! historically breaks — second-granular filesystems on macOS, mtimes zeroed by Docker
57//! layers, clock skew, fresh CI checkouts invalidating everything — and hashing a module
58//! costs microseconds against the milliseconds it saves.
59//!
60//! # Failure is a miss
61//!
62//! Every error path here returns "no entry" rather than propagating. A corrupt file, an
63//! unreadable directory, a store on a read-only filesystem: the module gets checked, as it
64//! would have been anyway. The one invariant worth stating is mypy's: an entry is written
65//! whole or not at all, via a temporary file and a rename, so a reader never sees half of
66//! one. Set `HTL_CACHE_DEBUG=1` to print why a lookup missed.
67//!
68//! # Why this lives in htl-core
69//!
70//! The store began in the CLI, which was the only reader. The linker is a reader too
71//! (#100): `htl build` and `include_bundle!` walk the require closure through
72//! [`crate::link`], and a module whose `gen` entry still holds — the same entry `htl test`
73//! writes and replays — has no reason to be generated again. `htl-cli` depends on `htl`,
74//! which depends on `htl-macros`, so a store the macros can open has to sit below both.
75//! The CLI keeps its flags, its `cache status` command and its report types; what moved is
76//! the store and the JSON shapes it writes, and the conversions between those shapes and
77//! [`crate::CheckInfo`], which every reader was carrying a copy of.
78//!
79//! Only `htl check` sweeps ([`Cache::sweep`]). It is the one reader that sees the whole
80//! project, so its keep-list is the whole project's; a build or a macro expansion sees one
81//! closure, and sweeping from that view would evict every other entry the project has.
82
83use anyhow::Result;
84use serde::{Deserialize, Serialize};
85use std::cell::RefCell;
86use std::collections::HashMap;
87use std::path::{Path, PathBuf};
88
89use crate::{CheckInfo, DependencyError, Fix, RequireSite};
90
91/// Bumped by hand when anything below changes shape. An entry stamped with a different
92/// value is a miss rather than an error: a fresh checkout and an upgrade both take that
93/// path in normal operation, which is why rustc treats its own header mismatch the same
94/// way.
95///
96/// 6: the stamp carries the checker's identity, and a `module` entry carries nothing
97/// else about the binary (#100), so the proc macros — whose `current_exe` is rustc, which
98/// does not change when htl does — still miss on a checker that no longer exists, and
99/// the CLI and the macros read each other's module entries.
100///
101/// 11: an entry's inputs are the module's require closure, and a [`Module`] carries the
102/// names each member of the closure required ([`Module::closure_requires`]), which the
103/// probe asks again (#400). An entry of 10 hashed only the direct requires and would hit
104/// over an edit two requires away — exactly the entries this change exists to stop
105/// replaying — so they are a miss rather than a default.
106///
107/// 12: an entry now carries the module's marker census ([`Module::markers`],
108/// [`CheckInfoJson::markers`]) — `htl adopt`'s input (#304). An entry of 11 has no such
109/// field, and a replay of one would give `htl adopt` nothing for a file the run did not
110/// check.
111///
112/// 13: an entry now also carries the module's construction-site census
113/// ([`Module::record_sites`], [`CheckInfoJson::record_sites`]) — `unmarked-struct`'s input
114/// and `htl adopt`'s `---@struct` row (#304). An entry of 12 has no such field, and
115/// `unmarked_structs` reads every file of the walk, replayed ones included, the same
116/// reason `markers` and `global_sites` carry through a replay.
117///
118/// 14: the census of 13 widened from construction sites alone to every site of a
119/// declared record, `as` casts included, and each [`RecordSiteJson`] now carries `kind`
120/// (`"literal"` / `"cast"`) and `sealed` beside what it already had — `unmarked-sealed`'s
121/// input (#304), beside `unmarked-struct`'s. An entry of 13 has neither field, and
122/// `unmarked_sealeds` reads every file of the walk the same way `unmarked_structs` does,
123/// so a replay of one would give it nothing for a file the run did not check.
124///
125/// A change to the Lua a `module` entry carries needs no bump: the stamp's `checker` is a
126/// hash of `prelude.lua` among the rest ([`crate::checker_identity`]), so a generator that
127/// emits different text is a different checker and every warm entry misses on its own.
128/// What this number is for is a change to the shape of what is stored — a field, a key, a
129/// meaning — which the hash cannot see.
130const FORMAT: u32 = 14;
131
132/// Where the store lives under the project root. Generated, and `htl init` puts `.htl/` in
133/// `.gitignore` — one line for the cache and the installed deps beside it, both
134/// machine-local and regenerated; an existing project adds the line by hand.
135const DIR: &str = ".htl/cache";
136
137/// How many entries a project's store may hold before a run starts dropping the ones it did
138/// not use.
139///
140/// Scaled to the project, because the natural size is one entry per module per way of
141/// invoking the check, and "a few ways" is what people do. The floor keeps a small project
142/// from evicting itself on its second invocation.
143fn default_bound(files: usize) -> usize {
144    (files * 4).max(256)
145}
146
147/// What produced an entry. A cache is only as good as its ability to notice it was
148/// written by something else.
149///
150/// The binary's length and modification time stand in for "which checker is this": during
151/// development the vendored Teal compiler, the prelude and the Rust side all change many
152/// times within one released version number, and a stamp that only carried the version
153/// would happily replay results from a checker that no longer exists. Rebuilding always
154/// moves both. This is the same reasoning behind sccache hashing the compiler binary into
155/// its key.
156///
157/// A [`MODULE`] entry — a required module's generated Lua — is stamped without the
158/// binary. Those entries are written by `htl test`, `htl build` and the proc macros, and
159/// inside a proc macro the binary is `rustc`: stamping with it would have the CLI and the
160/// macros overwrite each other's entries for the same module forever. What decides what a
161/// module generates is the Lua the checker is made of (the vendored `tl`, the prelude, the
162/// lints), so those entries carry [`crate::checker_identity`] — a hash of exactly that —
163/// beside the format and the htl version, and nothing that differs between two binaries
164/// built from the same sources.
165#[derive(Serialize, Deserialize, PartialEq, Eq, Debug)]
166struct Stamp {
167    format: u32,
168    htl: String,
169    checker: String,
170    exe_len: u64,
171    exe_mtime_ns: u128,
172}
173
174impl Stamp {
175    /// What this build stamps an entry of `kind` with: the binary too, except for
176    /// [`MODULE`] entries (see the type doc).
177    fn for_kind(kind: &str) -> Option<Self> {
178        if kind == MODULE {
179            Some(Self::portable())
180        } else {
181            Self::current()
182        }
183    }
184
185    fn current() -> Option<Self> {
186        let exe = std::env::current_exe().ok()?;
187        let m = std::fs::metadata(&exe).ok()?;
188        let mtime = m
189            .modified()
190            .ok()?
191            .duration_since(std::time::UNIX_EPOCH)
192            .ok()?
193            .as_nanos();
194        Some(Self {
195            exe_len: m.len(),
196            exe_mtime_ns: mtime,
197            ..Self::portable()
198        })
199    }
200
201    /// The stamp with nothing about the binary in it.
202    fn portable() -> Self {
203        Self {
204            format: FORMAT,
205            htl: env!("CARGO_PKG_VERSION").to_string(),
206            checker: crate::checker_identity().to_string(),
207            exe_len: 0,
208            exe_mtime_ns: 0,
209        }
210    }
211}
212
213/// One file the check read, by content.
214#[derive(Serialize, Deserialize, Debug)]
215struct Input {
216    path: String,
217    hash: String,
218}
219
220/// One directory a `require` could have resolved in, and what it offered *for the names
221/// this module asked for*.
222///
223/// This catches a file appearing where the checker would now find it, which no hash of the
224/// files it did read can see — the hole ccache documents in its direct mode. It used to
225/// hash every module name in the directory, which meant adding any file at all invalidated
226/// every module whose search path included it: writing one new module re-checked the whole
227/// project (#24). A file appearing under a name nobody requires cannot change what anybody
228/// resolved, so only the requested names go in.
229#[derive(Serialize, Deserialize, Debug)]
230struct Probe {
231    dir: String,
232    /// Hash of `(name, whether it resolves here)` over the names the module and the
233    /// members of its require closure required, sorted. Changing which directory a name
234    /// resolves in changes this for both directories involved, which is how a shadowing
235    /// file is caught. For the [`MODEL_PROBE`], the hash of the model's answer to each
236    /// question — a name, asked from the file that required it ([`Cache::questions`]).
237    names: String,
238}
239
240/// One question a probe asks again: a name, and the file that required it (`None` for the
241/// requires of a whole-run entry's modules, asked from nowhere in particular, as before
242/// the closure was probed).
243type Question = (Option<String>, String);
244
245/// The `dir` of the one probe a store with the project model records: not a directory,
246/// the model's answers ([`Cache::with_answers`]).
247const MODEL_PROBE: &str = "(project model)";
248
249/// The project model's answer for a module name, asked from a file (`None` for a whole
250/// run), as a string that changes whenever the answer does — which module, which files.
251/// What a store with the model probes instead of directories ([`Cache::with_answers`]).
252pub type Answers = std::sync::Arc<dyn Fn(Option<&Path>, &str) -> String + Send + Sync>;
253
254/// One diagnostic exactly as it was handed to the sink, so a replay goes through the same
255/// printing code the original run did rather than through a reconstruction of it.
256/// Reconstructing text from parsed fields is how a cache starts printing subtly different
257/// output from the run it claims to reproduce.
258#[derive(Serialize, Deserialize, Debug, Clone)]
259pub struct Recorded {
260    /// [`Severity::as_str`](crate::Severity::as_str)'s word. A string rather than the enum
261    /// because an entry outlives the build that wrote it, and a word this build does not
262    /// know has to read back as itself rather than fail the whole entry.
263    pub severity: String,
264    /// The diagnostic's parts. Its text is made from them by the one formatter
265    /// ([`crate::Diagnostic`]'s `Display`) at replay as it was at the original run, which is
266    /// why storing the parts reproduces the run: there is no second way to spell the line.
267    pub item: ItemJson,
268    /// The fix the diagnostic carried, for `htl fix` replaying instead of re-checking.
269    /// `None` when the diagnostic had none, which is most of them.
270    pub fix: Option<FixJson>,
271    /// Set when the text is an error in a module this one required rather than in this
272    /// one: the dependency and the file that required it. The entry carries every such
273    /// error the check found; the sink decides at replay, as it did at the original run,
274    /// which of them to say (once per run). Absent in entries written before this field
275    /// existed, which then read as having none.
276    #[serde(default, skip_serializing_if = "Option::is_none")]
277    pub dependency: Option<DependencyJson>,
278}
279
280/// A [`Fix`] as an entry (and the CLI's `--format json`) stores it.
281#[derive(Serialize, Deserialize, Debug, Clone)]
282pub struct FixJson {
283    /// `safe` / `unsafe` / `suggest`. Owned rather than `&'static str` because the store
284    /// reads these back, and a borrowed field cannot be deserialized into. The JSON is
285    /// unchanged either way.
286    pub applicability: String,
287    /// The edits, in the order [`Fix`] holds them. A fix is all of them or none: applying
288    /// part of one leaves the file in a state nobody asked for.
289    pub edits: Vec<EditJson>,
290}
291
292/// One replacement in a [`FixJson`]: a half-open span and what goes there.
293///
294/// Positions are the checker's — lines and columns counted from 1 — rather than byte
295/// offsets, because that is what the diagnostic beside it says and an entry that stored
296/// them differently would have to be trusted to convert the same way twice.
297#[derive(Serialize, Deserialize, Debug, Clone)]
298pub struct EditJson {
299    /// First line of the span.
300    pub line: usize,
301    /// First column of the span, inclusive.
302    pub col: usize,
303    /// Last line of the span; the same as `line` for an edit inside one line.
304    pub end_line: usize,
305    /// Column the span stops before, exclusive — so an empty span is an insertion.
306    pub end_col: usize,
307    /// What replaces the span. Empty to delete it.
308    pub text: String,
309}
310
311impl FixJson {
312    /// A fix as an entry stores it, for the run that writes one.
313    pub fn from_fix(f: &Fix) -> Self {
314        Self {
315            applicability: f.applicability.as_str().to_string(),
316            edits: f
317                .edits
318                .iter()
319                .map(|e| EditJson {
320                    line: e.line,
321                    col: e.col,
322                    end_line: e.end_line,
323                    end_col: e.end_col,
324                    text: e.text.clone(),
325                })
326                .collect(),
327        }
328    }
329
330    /// And back, for a replayed one. An applicability this build does not know reads as
331    /// the most cautious of the three rather than failing the entry — see the match below.
332    pub fn to_fix(&self) -> Fix {
333        Fix {
334            applicability: match self.applicability.as_str() {
335                "unsafe" => crate::Applicability::Unsafe,
336                "suggest" => crate::Applicability::Suggest,
337                // Anything else is a build that wrote a name this one does not know;
338                // treating it as the most cautious of the three is the only safe reading.
339                "safe" => crate::Applicability::Safe,
340                _ => crate::Applicability::Suggest,
341            },
342            edits: self
343                .edits
344                .iter()
345                .map(|e| crate::Edit {
346                    line: e.line,
347                    col: e.col,
348                    end_line: e.end_line,
349                    end_col: e.end_col,
350                    text: e.text.clone(),
351                })
352                .collect(),
353        }
354    }
355}
356
357/// What a dependency's diagnostic carries besides its text: the file that required it
358/// and where the file lives. Stored with the diagnostic ([`Recorded`]) so a replay says
359/// exactly what the run said, and decides the same way whether to say it.
360#[derive(Serialize, Deserialize, Debug, Clone)]
361pub struct DependencyJson {
362    /// The file the error is in, as the checker found it.
363    pub file: String,
364    /// The file whose `require` pulled it in — which is the one the reader is looking at,
365    /// and so the one a message about somebody else's file has to name.
366    pub required_by: String,
367    /// `dependency` / `external`, or none for a file of the project's own.
368    #[serde(default, skip_serializing_if = "Option::is_none")]
369    pub origin: Option<String>,
370}
371
372/// One literal `require` and where the checker resolved it. Kept because the project-level
373/// cycle lint runs over every file's requires, replayed ones included — a cycle that closes
374/// through a module nobody edited is still a cycle.
375#[derive(Serialize, Deserialize, Debug, Clone)]
376pub struct RequireJson {
377    /// The name as the source spells it.
378    pub module: String,
379    /// What it resolved to, `None` when the checker found nothing — a host module, or a
380    /// require that will fail. Stored either way: the cycle lint reads the resolved ones,
381    /// and a name that starts resolving is a change this entry has to notice.
382    pub path: Option<String>,
383    /// Where the `require` is, so a finding about it points at the call rather than the
384    /// file.
385    pub line: usize,
386    /// Column of the same.
387    pub col: usize,
388    /// [`RequireSite::erased`]: the value of a `local type` the generator drops. Stored so a
389    /// replayed module's requires say what the fresh check's would; absent in entries
390    /// written before the field existed, which is why the format was bumped rather than
391    /// defaulted — an old entry for an unchanged file would otherwise keep counting the
392    /// erased require as the host's.
393    #[serde(default)]
394    pub erased: bool,
395}
396
397/// What checking one module reported.
398#[derive(Serialize, Deserialize, Debug, Clone)]
399pub struct Module {
400    /// Its diagnostics, its contract lints included, in the order they were printed.
401    pub diagnostics: Vec<Recorded>,
402    /// How many of `diagnostics` were errors, counted when the entry was written.
403    ///
404    /// Stored rather than recounted on replay: the three are what a run adds up to decide
405    /// its exit code, and a replay that recounted them would be deciding that from its own
406    /// reading of the severity strings instead of from what the original run concluded.
407    pub errors: usize,
408    /// Warnings, as `errors`.
409    pub warnings: usize,
410    /// Lints, as `errors`.
411    pub lints: usize,
412    /// Every file the check read through `require`: the module's require closure
413    /// ([`CheckInfo::deps`]), not only its own requires. The next run keys on these, which
414    /// is how a dependency's edit invalidates its dependents — every one above it, however
415    /// many requires away.
416    pub deps: Vec<String>,
417    /// Every `require` in the source and where it went, which `deps` is the resolved,
418    /// deduplicated half of. Kept whole because the cycle lint reports a call site, and a
419    /// path cannot say which line asked for it.
420    pub requires: Vec<RequireJson>,
421    /// Every `global` declaration the check brought into scope and where
422    /// ([`CheckInfo::global_sites`]), kept for the same reason `requires` is: the
423    /// `global-redeclaration` lint runs over every file of the walk, replayed ones
424    /// included, and a declaration file is never a walked file, so a replayed module has
425    /// to carry the sites it saw. Absent in entries written before the field existed.
426    #[serde(default)]
427    pub global_sites: Vec<GlobalSiteJson>,
428    /// The marker census the check found on the file's declarations
429    /// ([`CheckInfo::markers`]), kept for the same reason `global_sites` is: `htl adopt`
430    /// (#304) reads every file of the walk, replayed ones included, and the module that
431    /// carries them is all a replay has. Absent in entries written before the field
432    /// existed.
433    #[serde(default)]
434    pub markers: Vec<MarkerSiteJson>,
435    /// The census of record sites the check found in the file
436    /// ([`CheckInfo::record_sites`]), kept for the same reason `markers` is: a run
437    /// replayed from the cache still feeds `unmarked_structs`, `unmarked_sealeds` and
438    /// `htl adopt`'s `---@struct` and `---@sealed` rows (#304), and the module that
439    /// carries them is all a replay has. Absent in entries written before the field
440    /// existed.
441    #[serde(default)]
442    pub record_sites: Vec<RecordSiteJson>,
443    /// The names each member of the require closure required — its require sites, resolved
444    /// or not ([`CheckInfo::closure_requires`]): the questions below the module that the
445    /// probe asks again, each from the member that asked it. `deps` says the files read are unchanged;
446    /// this is what says each name still means the file it meant. Entries written before the
447    /// field existed are a format behind (the stamp) and not read.
448    #[serde(default)]
449    pub closure_requires: Vec<ClosureRequireJson>,
450    /// The Lua this module generates, for entries under [`gen_key`]. `htl check` never needs
451    /// it and stores `None`; `htl test` stores it so a replay can go straight to running.
452    /// Absent when checking produced errors, since there is nothing to run then.
453    #[serde(default)]
454    pub code: Option<String>,
455    /// What checking reported, in the form the test runner needs it back.
456    ///
457    /// `htl check` replays `diagnostics` straight into the sink and never reconstructs a
458    /// `CheckInfo`; the test runner puts one into its report, so a replayed test file needs
459    /// the structured form rather than the printed one. Only `gen_key` entries carry it.
460    #[serde(default)]
461    pub check: Option<CheckInfoJson>,
462}
463
464/// A `CheckInfo` as an entry stores it.
465///
466/// Deliberately a separate type from the diagnostics `htl check` replays: those are text
467/// on their way to a terminal, these are the fields the runner reads back.
468#[derive(Serialize, Deserialize, Debug, Clone, Default)]
469pub struct CheckInfoJson {
470    /// [`CheckInfo::errors`], verbatim.
471    pub errors: Vec<String>,
472    /// [`CheckInfo::warnings`], verbatim.
473    pub warnings: Vec<String>,
474    /// [`CheckInfo::lints`], verbatim.
475    pub lints: Vec<String>,
476    /// [`CheckInfo::deps`] as strings. `PathBuf` does not round-trip through JSON on every
477    /// platform; these are normalised on the way in and rebuilt on the way out.
478    pub deps: Vec<String>,
479    /// [`CheckInfo::requires`], in this module's own JSON shape.
480    pub requires: Vec<RequireJson>,
481    /// Parallel to `errors`, as in [`CheckInfo::error_fixes`]: `error_fixes[i]` belongs to
482    /// `errors[i]`. Stored as a full-length vector of `Option` rather than as the fixes
483    /// that exist, so the pairing survives without an index.
484    pub error_fixes: Vec<Option<FixJson>>,
485    /// Parallel to `lints`, as `error_fixes` is to `errors`.
486    pub lint_fixes: Vec<Option<FixJson>>,
487    /// `CheckInfo::dependency_errors`, so the runner reads back the whole of what the
488    /// check said. Absent in entries written before the field existed.
489    #[serde(default)]
490    pub dependency_errors: Vec<DependencyErrorJson>,
491    /// [`CheckInfo::error_items`], [`CheckInfo::warning_items`] and
492    /// [`CheckInfo::lint_items`], in the same order as the text, the fixes left in
493    /// `error_fixes` / `lint_fixes` beside them. Entries written before these existed are
494    /// a format behind and are not read (the stamp's format).
495    #[serde(default)]
496    pub error_items: Vec<ItemJson>,
497    /// See `error_items`.
498    #[serde(default)]
499    pub warning_items: Vec<ItemJson>,
500    /// See `error_items`.
501    #[serde(default)]
502    pub lint_items: Vec<ItemJson>,
503    /// [`CheckInfo::syntax_errors`]. Absent in entries written before the field existed,
504    /// and read back as none — which they were: only a file that generated is stored with
505    /// its check, and a file the parser rejected generates nothing.
506    #[serde(default)]
507    pub syntax_errors: usize,
508    /// [`CheckInfo::global_sites`], so a run replayed from the cache still feeds the
509    /// `global-redeclaration` lint. Absent in entries written before the field existed;
510    /// those are a format behind (the stamp) and not read.
511    #[serde(default)]
512    pub global_sites: Vec<GlobalSiteJson>,
513    /// [`CheckInfo::markers`], so the test runner's report of a replayed module carries
514    /// the same census a fresh check would. Absent in entries written before the field
515    /// existed.
516    #[serde(default)]
517    pub markers: Vec<MarkerSiteJson>,
518    /// [`CheckInfo::record_sites`], so the test runner's report of a replayed module
519    /// carries the same census of record sites a fresh check would. Absent in entries
520    /// written before the field existed.
521    #[serde(default)]
522    pub record_sites: Vec<RecordSiteJson>,
523    /// [`CheckInfo::closure_requires`], so the check a replayed module hands back is the
524    /// whole of what the fresh one said.
525    #[serde(default)]
526    pub closure_requires: Vec<ClosureRequireJson>,
527}
528
529/// One edge below a module ([`CheckInfo::closure_requires`]) as an entry stores it: a member
530/// of the require closure and one name it required.
531#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
532pub struct ClosureRequireJson {
533    /// The member that required the name, as the checker found it.
534    pub from: String,
535    /// The name as that member's `require` spelt it.
536    pub module: String,
537}
538
539/// One [`crate::GlobalSite`] as an entry stores it.
540#[derive(Serialize, Deserialize, Debug, Clone)]
541pub struct GlobalSiteJson {
542    /// The declared name.
543    pub name: String,
544    /// The declaring file, as the checker found it.
545    pub file: String,
546    /// Line of the declared name.
547    pub line: usize,
548    /// Column of the same.
549    pub col: usize,
550}
551
552/// One [`crate::MarkerSite`] as an entry stores it.
553///
554/// `PartialEq` beyond what the neighbouring `*SiteJson` types derive: a replayed module's
555/// census is compared against a fresh check's (`crates/htl-core/tests/marker_census.rs`),
556/// which the other site types have had no reason to do yet.
557#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
558pub struct MarkerSiteJson {
559    /// The marker's name, without its `---@`.
560    pub marker: String,
561    /// What the marker is on: `"record"`, `"field"` or `"function"`.
562    pub kind: String,
563    /// Line of the declaration the marker is read from.
564    pub line: usize,
565    /// The declaration's name.
566    pub name: String,
567}
568
569/// One [`crate::RecordSite`] as an entry stores it.
570///
571/// `PartialEq`, for the same reason `MarkerSiteJson` carries it: a replayed module's
572/// census is compared against a fresh check's
573/// (`crates/htl-core/tests/record_sites.rs`).
574#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
575pub struct RecordSiteJson {
576    /// [`crate::RecordSite::record_file`].
577    pub record_file: String,
578    /// [`crate::RecordSite::record_line`].
579    pub record_line: usize,
580    /// [`crate::RecordSite::record_name`].
581    pub record_name: String,
582    /// [`crate::RecordSite::marked`].
583    pub marked: bool,
584    /// [`crate::RecordSite::line`].
585    pub line: usize,
586    /// [`crate::RecordSite::col`].
587    pub col: usize,
588    /// [`crate::RecordSite::kind`].
589    pub kind: String,
590    /// [`crate::RecordSite::sealed`].
591    pub sealed: bool,
592    /// [`crate::RecordSite::complete`].
593    pub complete: bool,
594}
595
596/// A [`crate::Diagnostic`]'s parts as an entry stores them: the position, the rule and the
597/// sentence. Its severity is the list it is in, and its fix is the one stored beside it.
598#[derive(Serialize, Deserialize, Debug, Clone, Default)]
599pub struct ItemJson {
600    /// [`Diagnostic::file`](crate::Diagnostic::file).
601    pub file: String,
602    /// [`Diagnostic::line`](crate::Diagnostic::line).
603    pub line: usize,
604    /// [`Diagnostic::col`](crate::Diagnostic::col).
605    pub col: usize,
606    /// [`Diagnostic::rule`](crate::Diagnostic::rule).
607    pub rule: Option<String>,
608    /// [`Diagnostic::message`](crate::Diagnostic::message).
609    pub message: String,
610}
611
612impl ItemJson {
613    /// `d`'s parts as an entry stores them.
614    pub fn from_diagnostic(d: &crate::Diagnostic) -> Self {
615        Self {
616            file: d.file.clone(),
617            line: d.line,
618            col: d.col,
619            rule: d.rule.clone(),
620            message: d.message.clone(),
621        }
622    }
623
624    /// The diagnostic these parts are, with `severity` and `fix`.
625    pub fn to_diagnostic(
626        &self,
627        severity: crate::Severity,
628        fix: Option<crate::Fix>,
629    ) -> crate::Diagnostic {
630        crate::Diagnostic {
631            severity,
632            file: self.file.clone(),
633            line: self.line,
634            col: self.col,
635            rule: self.rule.clone(),
636            message: self.message.clone(),
637            fix,
638            required_by: None,
639            origin: None,
640        }
641    }
642}
643
644impl CheckInfoJson {
645    /// What a check reported, in the form an entry stores it.
646    pub fn from_check(c: &CheckInfo) -> Self {
647        let items = |ds: &[crate::Diagnostic]| ds.iter().map(ItemJson::from_diagnostic).collect();
648        Self {
649            error_items: items(&c.error_items),
650            warning_items: items(&c.warning_items),
651            lint_items: items(&c.lint_items),
652            errors: c.errors.clone(),
653            syntax_errors: c.syntax_errors,
654            warnings: c.warnings.clone(),
655            lints: c.lints.clone(),
656            deps: c.deps.iter().map(|p| normal(p)).collect(),
657            requires: requires_json(c),
658            error_fixes: c
659                .error_fixes
660                .iter()
661                .map(|f| f.as_ref().map(FixJson::from_fix))
662                .collect(),
663            lint_fixes: c
664                .lint_fixes
665                .iter()
666                .map(|f| f.as_ref().map(FixJson::from_fix))
667                .collect(),
668            dependency_errors: c
669                .dependency_errors
670                .iter()
671                .map(|e| DependencyErrorJson {
672                    file: e.file.display().to_string(),
673                    required_by: e.required_by.display().to_string(),
674                    text: e.text.clone(),
675                    item: ItemJson::from_diagnostic(&e.diagnostic),
676                })
677                .collect(),
678            global_sites: global_sites_json(c),
679            markers: marker_sites_json(c),
680            record_sites: record_sites_json(c),
681            closure_requires: closure_requires_json(c),
682        }
683    }
684
685    /// And back, for a replayed module: the test runner puts it into its report, the
686    /// linker reads its lints, requires and dependency errors.
687    pub fn to_check(&self) -> CheckInfo {
688        let error_fixes: Vec<Option<crate::Fix>> = self
689            .error_fixes
690            .iter()
691            .map(|f| f.as_ref().map(FixJson::to_fix))
692            .collect();
693        let lint_fixes: Vec<Option<crate::Fix>> = self
694            .lint_fixes
695            .iter()
696            .map(|f| f.as_ref().map(FixJson::to_fix))
697            .collect();
698        let items = |list: &[ItemJson], severity, fixes: &[Option<crate::Fix>]| {
699            list.iter()
700                .enumerate()
701                .map(|(i, it)| it.to_diagnostic(severity, fixes.get(i).and_then(|f| f.clone())))
702                .collect()
703        };
704        CheckInfo {
705            error_items: items(&self.error_items, crate::Severity::Error, &error_fixes),
706            warning_items: items(&self.warning_items, crate::Severity::Warning, &[]),
707            lint_items: items(&self.lint_items, crate::Severity::Lint, &lint_fixes),
708            errors: self.errors.clone(),
709            syntax_errors: self.syntax_errors,
710            warnings: self.warnings.clone(),
711            lints: self.lints.clone(),
712            deps: self.deps.iter().map(PathBuf::from).collect(),
713            requires: requires_from_json(&self.requires),
714            error_fixes,
715            lint_fixes,
716            dependency_errors: self
717                .dependency_errors
718                .iter()
719                .map(|e| DependencyError {
720                    file: PathBuf::from(&e.file),
721                    required_by: PathBuf::from(&e.required_by),
722                    text: e.text.clone(),
723                    diagnostic: e.item.to_diagnostic(crate::Severity::Error, None),
724                })
725                .collect(),
726            global_sites: global_sites_from_json(&self.global_sites),
727            markers: marker_sites_from_json(&self.markers),
728            record_sites: record_sites_from_json(&self.record_sites),
729            closure_requires: closure_requires_from_json(&self.closure_requires),
730            types: Default::default(),
731        }
732    }
733}
734
735/// One `htl::DependencyError` as an entry stores it.
736#[derive(Serialize, Deserialize, Debug, Clone)]
737pub struct DependencyErrorJson {
738    /// The required module the error is in.
739    pub file: String,
740    /// The file whose `require` reached it — the one being checked, and the one a report
741    /// names.
742    pub required_by: String,
743    /// The error as the dependency's own check phrased it.
744    pub text: String,
745    /// The same error in its parts ([`crate::DependencyError::diagnostic`]).
746    #[serde(default)]
747    pub item: ItemJson,
748}
749
750/// `CheckInfo`'s requires in the form an entry stores them.
751pub fn requires_json(c: &CheckInfo) -> Vec<RequireJson> {
752    c.requires
753        .iter()
754        .map(|r| RequireJson {
755            module: r.module.clone(),
756            path: r.path.as_ref().map(|p| p.to_string_lossy().into_owned()),
757            line: r.line,
758            col: r.col,
759            erased: r.erased,
760        })
761        .collect()
762}
763
764/// [`CheckInfo::global_sites`] in the shape an entry stores.
765pub fn global_sites_json(c: &CheckInfo) -> Vec<GlobalSiteJson> {
766    c.global_sites
767        .iter()
768        .map(|s| GlobalSiteJson {
769            name: s.name.clone(),
770            file: s.file.display().to_string(),
771            line: s.line,
772            col: s.col,
773        })
774        .collect()
775}
776
777/// [`CheckInfo::markers`] in the shape an entry stores.
778pub fn marker_sites_json(c: &CheckInfo) -> Vec<MarkerSiteJson> {
779    c.markers
780        .iter()
781        .map(|s| MarkerSiteJson {
782            marker: s.marker.clone(),
783            kind: s.kind.clone(),
784            line: s.line,
785            name: s.name.clone(),
786        })
787        .collect()
788}
789
790/// [`CheckInfo::record_sites`] in the shape an entry stores.
791pub fn record_sites_json(c: &CheckInfo) -> Vec<RecordSiteJson> {
792    c.record_sites
793        .iter()
794        .map(|s| RecordSiteJson {
795            record_file: s.record_file.clone(),
796            record_line: s.record_line,
797            record_name: s.record_name.clone(),
798            marked: s.marked,
799            line: s.line,
800            col: s.col,
801            kind: s.kind.clone(),
802            sealed: s.sealed,
803            complete: s.complete,
804        })
805        .collect()
806}
807
808/// [`CheckInfo::closure_requires`] in the shape an entry stores.
809pub fn closure_requires_json(c: &CheckInfo) -> Vec<ClosureRequireJson> {
810    c.closure_requires
811        .iter()
812        .map(|(from, module)| ClosureRequireJson {
813            from: normal(from),
814            module: module.clone(),
815        })
816        .collect()
817}
818
819fn closure_requires_from_json(edges: &[ClosureRequireJson]) -> Vec<(PathBuf, String)> {
820    edges
821        .iter()
822        .map(|e| (PathBuf::from(&e.from), e.module.clone()))
823        .collect()
824}
825
826fn global_sites_from_json(sites: &[GlobalSiteJson]) -> Vec<crate::GlobalSite> {
827    sites
828        .iter()
829        .map(|s| crate::GlobalSite {
830            name: s.name.clone(),
831            file: PathBuf::from(&s.file),
832            line: s.line,
833            col: s.col,
834        })
835        .collect()
836}
837
838fn marker_sites_from_json(sites: &[MarkerSiteJson]) -> Vec<crate::MarkerSite> {
839    sites
840        .iter()
841        .map(|s| crate::MarkerSite {
842            marker: s.marker.clone(),
843            kind: s.kind.clone(),
844            line: s.line,
845            name: s.name.clone(),
846        })
847        .collect()
848}
849
850fn record_sites_from_json(sites: &[RecordSiteJson]) -> Vec<crate::RecordSite> {
851    sites
852        .iter()
853        .map(|s| crate::RecordSite {
854            record_file: s.record_file.clone(),
855            record_line: s.record_line,
856            record_name: s.record_name.clone(),
857            marked: s.marked,
858            line: s.line,
859            col: s.col,
860            kind: s.kind.clone(),
861            sealed: s.sealed,
862            complete: s.complete,
863        })
864        .collect()
865}
866
867fn requires_from_json(requires: &[RequireJson]) -> Vec<RequireSite> {
868    requires
869        .iter()
870        .map(|r| RequireSite {
871            module: r.module.clone(),
872            path: r.path.as_ref().map(PathBuf::from),
873            line: r.line,
874            col: r.col,
875            erased: r.erased,
876        })
877        .collect()
878}
879
880impl Module {
881    /// A `CheckInfo` carrying only what the project-level lints read.
882    ///
883    /// `require_cycles` runs over every file in the walk, replayed ones included — a cycle
884    /// that closes through a module nobody edited is still a cycle — so a replayed module
885    /// has to produce something that lint can read. `global_redeclarations` reads the
886    /// sites the same way, and the error texts, for the one error of the checker's it
887    /// defers to; the recorded diagnostics give those back. `unmarked_structs` and
888    /// `unmarked_sealeds` read `record_sites` the same way again: a record's
889    /// construction and cast sites are typically in files other than its own
890    /// declaration, so a project-level census that skipped a replayed module would
891    /// silently lose every site that module built or cast. Everything is already
892    /// printed by then, and nothing downstream looks at the other fields.
893    pub fn requires_only(&self) -> CheckInfo {
894        CheckInfo {
895            deps: self.deps.iter().map(PathBuf::from).collect(),
896            requires: requires_from_json(&self.requires),
897            global_sites: global_sites_from_json(&self.global_sites),
898            record_sites: record_sites_from_json(&self.record_sites),
899            closure_requires: closure_requires_from_json(&self.closure_requires),
900            errors: self
901                .diagnostics
902                .iter()
903                .filter(|r| r.severity == crate::Severity::Error.as_str())
904                .map(|r| r.item.message.clone())
905                .collect(),
906            ..Default::default()
907        }
908    }
909
910    /// A `gen` entry as the linker and the test runner write it: no printed diagnostics
911    /// (they print from the structured form), the generated Lua, and what the check said.
912    pub fn generated(c: &CheckInfo, code: String) -> Self {
913        Self {
914            diagnostics: Vec::new(),
915            errors: c.errors.len(),
916            warnings: c.warnings.len(),
917            lints: c.lints.len(),
918            deps: c.deps.iter().map(|p| normal(p)).collect(),
919            requires: requires_json(c),
920            global_sites: global_sites_json(c),
921            markers: marker_sites_json(c),
922            record_sites: record_sites_json(c),
923            closure_requires: closure_requires_json(c),
924            code: Some(code),
925            check: Some(CheckInfoJson::from_check(c)),
926        }
927    }
928}
929
930/// The directories an entry for `file` probes when there is no project model: the file's
931/// own, the one place a file outside any project resolves its `require`s in
932/// (`project::file_view`).
933///
934/// A store with the model probes the model's answers instead
935/// ([`Cache::with_answers`]), and never reads these. Listed whether or not it exists: a
936/// module appearing beside the file after an entry was written changes what a name
937/// resolves to while every file the entry recorded still hashes the same — the hole
938/// ccache documents in its direct mode.
939pub fn search_dirs(file: &Path) -> Vec<PathBuf> {
940    file.parent().map(Path::to_path_buf).into_iter().collect()
941}
942
943#[derive(Serialize, Deserialize, Debug)]
944struct Entry {
945    stamp: Stamp,
946    /// The file this entry is about, and which question it answers.
947    ///
948    /// Neither is needed to use the entry — the key already decided both. They are here so
949    /// the store can be described: the files on disk are named by hash, and a cache nobody
950    /// can read the contents of is one nobody can reason about (`htl cache status`).
951    subject: String,
952    kind: String,
953    inputs: Vec<Input>,
954    probes: Vec<Probe>,
955    module: Module,
956}
957
958/// One entry holding the whole walk, for `Mode::WholeRun`.
959#[derive(Serialize, Deserialize, Debug)]
960struct RunEntry {
961    stamp: Stamp,
962    /// The files the walk visited, and which question this answers. See `Entry`.
963    subjects: Vec<String>,
964    kind: String,
965    inputs: Vec<Input>,
966    probes: Vec<Probe>,
967    /// One per file in the walk, in walk order. A different count means the walk itself
968    /// changed, which is a miss before any hash is compared.
969    modules: Vec<Module>,
970}
971
972/// One entry holding what `lshape-version` found for the project's declaration and runtime.
973#[derive(Serialize, Deserialize, Debug)]
974struct LshapeEntry {
975    stamp: Stamp,
976    /// The declaration and runtime files this entry read, and which question it answers.
977    /// See `Entry`.
978    subjects: Vec<String>,
979    kind: String,
980    inputs: Vec<Input>,
981    probes: Vec<Probe>,
982    diagnostics: Vec<Recorded>,
983}
984
985/// How much of a run one entry covers.
986///
987/// The two are a real trade rather than one being a refinement of the other, and which
988/// wins depends on where in the dependency graph the edit lands. What `PerModule` saves
989/// depends entirely on that: editing a leaf replays everything else, and editing the
990/// module at the bottom of the graph saves nothing, because everything above it has to be
991/// checked again anyway. Neither mode is slower than a cold check. `WholeRun` keeps one
992/// entry per invocation rather than one per module, which is the reason to reach for it
993/// if the number of files in `.htl/` is the problem. `PerModule` is the default because
994/// editing is the common case.
995#[derive(Copy, Clone, PartialEq, Eq, Debug, Default)]
996pub enum Mode {
997    /// One entry per module. An edit costs the module, its dependents, and what those pull
998    /// in; everything else replays.
999    #[default]
1000    PerModule,
1001    /// One entry for the walk. Any edit anywhere re-checks everything, but a run where
1002    /// nothing moved reads one file instead of one per module.
1003    WholeRun,
1004}
1005
1006impl Mode {
1007    /// The mode a flag or an environment variable names, or `None` for a word this build
1008    /// does not know — which the caller reports as a bad argument rather than silently
1009    /// taking the default for.
1010    pub fn parse(s: &str) -> Option<Self> {
1011        match s {
1012            "per-module" => Some(Mode::PerModule),
1013            "whole-run" => Some(Mode::WholeRun),
1014            _ => None,
1015        }
1016    }
1017
1018    /// The word [`parse`](Self::parse) reads, and the one a run's summary line prints. The
1019    /// two are one spelling on purpose, so what a report says can be pasted back as a flag.
1020    pub fn as_str(self) -> &'static str {
1021        match self {
1022            Mode::PerModule => "per-module",
1023            Mode::WholeRun => "whole-run",
1024        }
1025    }
1026}
1027
1028/// Everything the store needs from outside itself.
1029///
1030/// Built once, by the caller that is already reading flags and the environment, and passed
1031/// in. Nothing in this module reaches for `std::env`: a run's behaviour is decided in one
1032/// place, so reading that place tells you what the run will do.
1033#[derive(Copy, Clone, Debug)]
1034pub struct Options {
1035    /// `--no-cache` turns this off. Separate from how the cache is grained.
1036    pub enabled: bool,
1037    /// How much of a run one entry covers. Held here rather than decided per lookup, so
1038    /// the whole run reads and writes the same grain.
1039    pub mode: Mode,
1040    /// Say why lookups missed, and what the run did with the store.
1041    pub explain: bool,
1042    /// Entries a project's store may hold. `None` scales it to the project size.
1043    pub max_entries: Option<usize>,
1044}
1045
1046impl Default for Options {
1047    fn default() -> Self {
1048        Self {
1049            enabled: true,
1050            mode: Mode::default(),
1051            explain: false,
1052            max_entries: None,
1053        }
1054    }
1055}
1056
1057/// What one run did with the store.
1058///
1059/// Kept by the store rather than by its callers, so a report cannot claim a number the store
1060/// disagrees with.
1061#[derive(Default, Copy, Clone, Debug)]
1062pub struct Stats {
1063    /// Modules replayed from an entry.
1064    pub hits: usize,
1065    /// Modules the store had nothing usable for, so they were checked. Counts a key that
1066    /// was not there and an entry whose inputs had moved alike — both cost the same check,
1067    /// and `explain` is what tells them apart.
1068    pub misses: usize,
1069    /// Entries written. Not the same as `misses`: a run with writing off, or one whose
1070    /// write failed, misses without storing.
1071    pub stored: usize,
1072    /// Entries the sweep dropped, which only `htl check` does.
1073    pub evicted: usize,
1074}
1075
1076impl Stats {
1077    /// One line, for the end of a run. `None` when the run did not touch the store at all,
1078    /// which is worth saying by saying nothing.
1079    pub fn summary(&self, mode: Mode) -> Option<String> {
1080        let touched = self.hits + self.misses + self.stored + self.evicted;
1081        (touched > 0).then(|| {
1082            format!(
1083                "htl cache: {} hit, {} missed, {} written, {} evicted ({})",
1084                self.hits,
1085                self.misses,
1086                self.stored,
1087                self.evicted,
1088                mode.as_str()
1089            )
1090        })
1091    }
1092}
1093
1094/// The identity of one module under one invocation: same key, same question.
1095///
1096/// Carries which question it is, so an entry written under it can record that too — the file
1097/// on disk is named by the hash, and a store whose contents cannot be described is one
1098/// nobody can reason about.
1099#[derive(Debug, Clone)]
1100pub struct Key {
1101    hash: String,
1102    kind: &'static str,
1103}
1104
1105/// Hash of a file's contents, or `None` if it cannot be read — an unreadable input is a
1106/// miss, never a silent hit.
1107fn hash_file(p: &Path) -> Option<String> {
1108    let bytes = std::fs::read(p).ok()?;
1109    Some(blake3::hash(&bytes).to_hex().to_string())
1110}
1111
1112/// A path in the form it is stored and compared in: absolute and symlink-resolved where
1113/// possible.
1114///
1115/// Normalizing at one producer and not another is a real bug rather than a tidiness
1116/// concern: mypy shipped a version where a file checked directly recorded a relative path
1117/// while the same file reached as an import recorded an absolute one, so its hash depended
1118/// on how it was reached. Everything that goes into an entry comes through here.
1119pub fn normal(p: &Path) -> String {
1120    std::fs::canonicalize(p)
1121        .unwrap_or_else(|_| p.to_path_buf())
1122        .to_string_lossy()
1123        .into_owned()
1124}
1125
1126/// Whether `name` resolves to a file in `dir`.
1127///
1128/// The shapes are the ones `H.add_path` puts on the search path (`dir/?.tl`,
1129/// `dir/?/init.tl`) plus the declaration and Lua forms the checker falls back to. A dot in
1130/// a module name is a directory separator, as it is for Lua's own searcher.
1131///
1132/// This is an existence question, not a resolution: it says the name *could* be found here,
1133/// and comparing the answer across every directory on the path is what makes a change of
1134/// resolution visible. Getting it wrong in the direction of "present" costs a re-check;
1135/// there is no direction in which it produces a wrong answer.
1136fn resolves_in(dir: &Path, name: &str) -> bool {
1137    let stem = name.replace('.', "/");
1138    ["tl", "d.tl", "lua"]
1139        .iter()
1140        .any(|ext| dir.join(format!("{stem}.{ext}")).is_file())
1141        || ["init.tl", "init.d.tl", "init.lua"]
1142            .iter()
1143            .any(|f| dir.join(&stem).join(f).is_file())
1144}
1145
1146/// The key for one module in this invocation.
1147///
1148/// `spelling` is the path as the CLI will hand it to the checker, not a resolved one: the
1149/// checker echoes it into every diagnostic it prints, so `htl check .` and
1150/// `htl check src/a.tl` produce different text for the same module and cannot share an
1151/// entry. The working directory completes it, since the same relative spelling means a
1152/// different file from somewhere else.
1153pub fn module_key(spelling: &Path, lint: Option<&str>) -> Key {
1154    key_with(CHECK, spelling, lint)
1155}
1156
1157/// An entry holding what checking a module reported.
1158pub const CHECK: &str = "check";
1159/// An entry holding that, plus the Lua the module generates.
1160pub const GEN: &str = "gen";
1161
1162/// The key for one test file's checked-and-generated form.
1163///
1164/// The same material as [`module_key`] under a different prefix, so the two never collide.
1165/// They are separate entries on purpose: a `htl check` entry holds diagnostics and nothing
1166/// else, and making it carry a few KB of generated Lua would cost every check run a parse it
1167/// has no use for.
1168pub fn gen_key(spelling: &Path, lint: Option<&str>) -> Key {
1169    key_with(GEN, spelling, lint)
1170}
1171
1172/// An entry holding a required module's generated Lua, keyed by the file it is.
1173pub const MODULE: &str = "module";
1174
1175/// The key for a module's checked-and-generated form, by the file it is.
1176///
1177/// A test file is keyed by its spelling ([`gen_key`]) because the runner prints that
1178/// spelling. A module reached through `require` is another matter: the test runner meets
1179/// it by the path the checker resolved, the linker by the one it resolved, `htl build` and
1180/// `include_bundle!` from different working directories — and its generated Lua is the
1181/// same file's Lua whichever way it was reached. So the key is the canonical path and the
1182/// lint selection, and nothing about the invocation, and the entry is stamped without the
1183/// binary (the entry's stamp): that is what lets `htl test`, `htl build` and the macros replay one
1184/// another's entries (#100).
1185pub fn module_gen_key(path: &Path, lint: Option<&str>) -> Key {
1186    let mut h = blake3::Hasher::new();
1187    h.update(MODULE.as_bytes());
1188    h.update(b"\0");
1189    h.update(normal(path).as_bytes());
1190    h.update(b"\0");
1191    h.update(lint.unwrap_or("").as_bytes());
1192    Key {
1193        hash: h.finalize().to_hex().to_string(),
1194        kind: MODULE,
1195    }
1196}
1197
1198/// Whether a module's source could `require` anything: the token appears in it.
1199///
1200/// `gen_lua` comes back without requires for a module the checker already holds — it
1201/// serves the generated code without walking the file again — and a reader that needs the
1202/// requires (the linker's walk, an entry's validation) asks `check` for them. A full check
1203/// for every leaf module is the wrong price for that, and a file that never says
1204/// `require` has nothing to ask about. Word-bounded, so `required_by` does not count.
1205pub fn source_mentions_require(text: &str) -> bool {
1206    let bytes = text.as_bytes();
1207    let ident = |b: u8| b.is_ascii_alphanumeric() || b == b'_';
1208    text.match_indices("require").any(|(i, _)| {
1209        let before = i.checked_sub(1).map(|j| bytes[j]);
1210        let after = bytes.get(i + "require".len()).copied();
1211        !before.is_some_and(ident) && !after.is_some_and(ident)
1212    })
1213}
1214
1215/// The switches every reader of the store honours, from the environment:
1216/// `HTL_NO_CACHE` (neither read nor write), `HTL_CACHE_DEBUG` (say why lookups missed),
1217/// `HTL_CACHE_MAX_ENTRIES` (the sweep's bound; the test suite's, since nothing else can
1218/// reach a few hundred entries by honest means). A command's flags override what this
1219/// read; the proc macros have no flags and take this as it is.
1220impl Options {
1221    /// The options the environment asks for, before any flag is applied.
1222    pub fn from_env() -> Self {
1223        Self {
1224            enabled: std::env::var_os("HTL_NO_CACHE").is_none(),
1225            mode: Mode::PerModule,
1226            explain: std::env::var_os("HTL_CACHE_DEBUG").is_some(),
1227            max_entries: std::env::var_os("HTL_CACHE_MAX_ENTRIES")
1228                .and_then(|v| v.to_str().and_then(|s| s.parse().ok())),
1229        }
1230    }
1231}
1232
1233/// Why nothing may be written under `root`, if nothing may — build scratch, named.
1234///
1235/// A macro expands wherever cargo compiles the crate: in the checkout, but also in the
1236/// copy `cargo publish` verifies under `target/package/<crate>/` — where a new file makes
1237/// cargo abort with "Source directory was modified" — and in a registry checkout under
1238/// `$CARGO_HOME/registry/src`, which nothing should write to. Both are recognisable by
1239/// their path, and neither is anybody's tree to keep anything in.
1240///
1241/// **The rule is the whole `.htl/`, not the store.** Every write htl makes as a side
1242/// effect of reading a project asks here first, and there are two: the run cache
1243/// (`project::store_refusal`, which every reader of the store comes through) and the
1244/// entry links ([`crate::pkg::MluaProject::link_entries`], through
1245/// [`Htl::apply_project`](crate::Htl::apply_project)). The link repair was outside the
1246/// rule until #267, where it wrote `.htl/modules/entries/<dep>` into the tree `cargo
1247/// package` had just built and cargo refused the tarball. A reader of a scratch tree
1248/// reads it exactly as it was packaged.
1249///
1250/// **`target` is recognised by the path and nothing else.** cargo publishes no signal
1251/// saying "this build is the verification step", so htl reads the convention instead: a
1252/// path component named `target`. That has two costs, both accepted for having one rule
1253/// in one function rather than one per writer. A `CARGO_TARGET_DIR` pointing outside the
1254/// tree defeats it — the verify copy is then not under a `target` directory, and htl
1255/// writes in it as it would in a checkout. And a project that keeps its own sources under
1256/// a directory called `target` goes without a store it could have had; a miss costs a
1257/// check, which is the cheaper way to be wrong.
1258pub fn scratch_root(root: &Path) -> Option<&'static str> {
1259    if root.components().any(|c| c.as_os_str() == "target") {
1260        return Some("under a `target` directory");
1261    }
1262    let cargo_home = std::env::var_os("CARGO_HOME")
1263        .map(PathBuf::from)
1264        .or_else(|| std::env::var_os("HOME").map(|h| PathBuf::from(h).join(".cargo")));
1265    if let Some(registry) = cargo_home.map(|h| h.join("registry"))
1266        && root.starts_with(&registry)
1267    {
1268        return Some("a registry checkout");
1269    }
1270    None
1271}
1272
1273fn key_with(kind: &'static str, spelling: &Path, lint: Option<&str>) -> Key {
1274    let mut h = blake3::Hasher::new();
1275    h.update(kind.as_bytes());
1276    h.update(b"\0");
1277    h.update(spelling.to_string_lossy().as_bytes());
1278    h.update(b"\0");
1279    h.update(lint.unwrap_or("").as_bytes());
1280    h.update(b"\0");
1281    h.update(cwd().as_bytes());
1282    Key {
1283        hash: h.finalize().to_hex().to_string(),
1284        kind,
1285    }
1286}
1287
1288/// The key for the walk as a whole, under `Mode::WholeRun`.
1289///
1290/// Every file's spelling goes in, in order: a walk that visits a different set, or the same
1291/// set named differently, is a different run and prints different text.
1292pub fn run_key(files: &[PathBuf], lint: Option<&str>) -> Key {
1293    let mut h = blake3::Hasher::new();
1294    h.update(b"htl-run\0");
1295    for f in files {
1296        h.update(f.to_string_lossy().as_bytes());
1297        h.update(b"\0");
1298    }
1299    h.update(lint.unwrap_or("").as_bytes());
1300    h.update(b"\0");
1301    h.update(cwd().as_bytes());
1302    Key {
1303        hash: h.finalize().to_hex().to_string(),
1304        kind: RUN,
1305    }
1306}
1307
1308/// One entry covering a whole walk (`Mode::WholeRun`).
1309pub const RUN: &str = "run";
1310
1311/// One entry holding what `lshape-version` reported for the project's declaration and
1312/// runtime (#531).
1313pub const LSHAPE: &str = "lshape";
1314
1315/// The key for the project's `lshape-version` verdict.
1316///
1317/// Carries the working directory (findings spell their paths against it) and `LUA_PATH`
1318/// (a declaration with no runtime in the model resolves its runtime on `package.path`,
1319/// #530).
1320pub fn lshape_key() -> Key {
1321    let mut h = blake3::Hasher::new();
1322    h.update(LSHAPE.as_bytes());
1323    h.update(b"\0");
1324    h.update(cwd().as_bytes());
1325    h.update(b"\0");
1326    if let Some(lp) = std::env::var_os("LUA_PATH") {
1327        h.update(lp.as_encoded_bytes());
1328    }
1329    Key {
1330        hash: h.finalize().to_hex().to_string(),
1331        kind: LSHAPE,
1332    }
1333}
1334
1335/// The working directory, resolved once. This is called for every module in the walk, and
1336/// resolving it means a `canonicalize` syscall — doing that per module is a measurable part
1337/// of a run that replays everything and does nothing else.
1338fn cwd() -> &'static str {
1339    static CWD: std::sync::OnceLock<String> = std::sync::OnceLock::new();
1340    CWD.get_or_init(|| {
1341        std::env::current_dir()
1342            .map(|p| normal(&p))
1343            .unwrap_or_default()
1344    })
1345}
1346
1347/// One entry, as `htl cache status` describes it.
1348#[derive(Serialize, Debug)]
1349pub struct EntrySummary {
1350    /// `check`, `gen`, `module`, `run`, `lshape` — or `unreadable` for a file this build
1351    /// cannot parse, which is worth showing rather than hiding, since it is also a permanent
1352    /// miss.
1353    pub kind: String,
1354    /// The files it is about. One, except for a whole-run entry.
1355    pub subjects: Vec<String>,
1356    /// Size of the entry's file on disk, which is what `htl cache status` totals.
1357    pub bytes: u64,
1358    /// Seconds since the entry was last written or replayed. The sweep drops the oldest.
1359    pub age_secs: u64,
1360}
1361
1362/// What a project's store holds.
1363#[derive(Serialize, Debug)]
1364pub struct Contents {
1365    /// The store's directory, said even when it holds nothing — "empty" and "not where you
1366    /// thought" are different answers and a reader cannot tell them apart without it.
1367    pub dir: String,
1368    /// One per entry file. Empty when the directory is missing or unreadable, which are
1369    /// not distinguished: neither is a store this run can use.
1370    pub entries: Vec<EntrySummary>,
1371    /// The whole store's bytes, summed over `entries`.
1372    pub bytes: u64,
1373}
1374
1375/// Read the store and say what is in it.
1376///
1377/// Every entry is parsed, which is why this is a command rather than something a run does on
1378/// the side. Nothing here validates: an entry listed may still be a miss on the next run
1379/// because its inputs moved. This answers "what is stored", not "what would be reused".
1380pub fn describe(root: &Path) -> Contents {
1381    let dir = root.join(DIR);
1382    let mut out = Contents {
1383        dir: dir.to_string_lossy().into_owned(),
1384        entries: Vec::new(),
1385        bytes: 0,
1386    };
1387    let Ok(rd) = std::fs::read_dir(&dir) else {
1388        return out;
1389    };
1390    let now = std::time::SystemTime::now();
1391    for e in rd.flatten() {
1392        let path = e.path();
1393        if path.extension().is_none_or(|x| x != "json") {
1394            continue;
1395        }
1396        let Ok(meta) = e.metadata() else { continue };
1397        out.bytes += meta.len();
1398        let age_secs = meta
1399            .modified()
1400            .ok()
1401            .and_then(|t| now.duration_since(t).ok())
1402            .map(|d| d.as_secs())
1403            .unwrap_or(0);
1404        let v: serde_json::Value = std::fs::read_to_string(&path)
1405            .ok()
1406            .and_then(|raw| serde_json::from_str(&raw).ok())
1407            .unwrap_or(serde_json::Value::Null);
1408        let kind = v
1409            .get("kind")
1410            .and_then(|k| k.as_str())
1411            .unwrap_or("unreadable")
1412            .to_string();
1413        let subjects = match (v.get("subject"), v.get("subjects")) {
1414            (Some(s), _) if s.is_string() => vec![s.as_str().unwrap_or_default().to_string()],
1415            (_, Some(a)) if a.is_array() => a
1416                .as_array()
1417                .unwrap()
1418                .iter()
1419                .filter_map(|x| x.as_str().map(str::to_string))
1420                .collect(),
1421            _ => Vec::new(),
1422        };
1423        out.entries.push(EntrySummary {
1424            kind,
1425            subjects,
1426            bytes: meta.len(),
1427            age_secs,
1428        });
1429    }
1430    out.entries
1431        .sort_by(|a, b| a.kind.cmp(&b.kind).then(a.subjects.cmp(&b.subjects)));
1432    out
1433}
1434
1435/// A store rooted at a project.
1436pub struct Cache {
1437    dir: PathBuf,
1438    opts: Options,
1439    stats: RefCell<Stats>,
1440    /// Content hashes taken during this run, by normalized path.
1441    ///
1442    /// Entries overlap heavily: every module in a project tends to require the same few
1443    /// leaves, so a leaf appears in the inputs of most entries. Without this, checking 48
1444    /// modules against the store reads and hashes the shared ones 48 times — which is most
1445    /// of what a fully replayed run spends. A file is assumed not to change while one
1446    /// `htl check` is running; if it does, the run's answer was undefined anyway.
1447    hashes: RefCell<HashMap<String, Option<String>>>,
1448    /// Name-resolves-here answers taken during this run, by (directory, name). Same
1449    /// reasoning.
1450    dirs: RefCell<HashMap<(String, String), bool>>,
1451    /// The project model's answers, when the caller has a model ([`with_answers`](Self::with_answers)).
1452    answers: Option<Answers>,
1453}
1454
1455impl Cache {
1456    /// `None` when the caller turned caching off, or when this build cannot identify
1457    /// itself well enough to be sure an entry is its own.
1458    pub fn open(root: &Path, opts: Options) -> Option<Self> {
1459        if !opts.enabled {
1460            return None;
1461        }
1462        Stamp::current()?;
1463        Some(Self {
1464            dir: root.join(DIR),
1465            opts,
1466            stats: RefCell::new(Stats::default()),
1467            hashes: RefCell::new(HashMap::new()),
1468            dirs: RefCell::new(HashMap::new()),
1469            answers: None,
1470        })
1471    }
1472
1473    /// Probe with the project model's `answers` instead of the search path's directories.
1474    ///
1475    /// The directories were a stand-in for the question an entry has to ask — does each
1476    /// name the module required still mean the file it meant — and a list of them could
1477    /// only ask it of the directories on the list: a helper added directly under `tests/`
1478    /// was not seen by a test in `tests/sub/`, whose list had `tests/sub` and not `tests`.
1479    /// The model answers the question itself. An entry recorded this way misses in a run
1480    /// without the model, and the reverse, since neither can check the other's probe.
1481    pub fn with_answers(mut self, answers: Answers) -> Self {
1482        self.answers = Some(answers);
1483        self
1484    }
1485
1486    /// The model's answer to each question — a name, asked from the file that required it
1487    /// (`None`: from nowhere in particular, the whole-run entry's own requires) — hashed.
1488    /// The asker is hashed beside the name: the same name asked from two files is two
1489    /// questions, and the model can answer them differently.
1490    fn model_hash(&self, answers: &Answers, questions: &[Question]) -> String {
1491        let mut h = blake3::Hasher::new();
1492        for (asker, n) in questions {
1493            h.update(asker.as_deref().unwrap_or("").as_bytes());
1494            h.update(b"\0");
1495            h.update(n.as_bytes());
1496            h.update(b"\0");
1497            h.update(answers(asker.as_deref().map(Path::new), n).as_bytes());
1498            h.update(b"\x01");
1499        }
1500        h.finalize().to_hex().to_string()
1501    }
1502
1503    /// What this run has done with the store so far.
1504    pub fn stats(&self) -> Stats {
1505        *self.stats.borrow()
1506    }
1507
1508    /// Say why something was not reused. Only when asked; the caller decided that once.
1509    fn miss(&self, reason: &str) {
1510        self.stats.borrow_mut().misses += 1;
1511        if self.opts.explain {
1512            eprintln!("htl cache: miss ({reason})");
1513        }
1514    }
1515
1516    /// The hash of a file, taken once per run.
1517    fn hash_of(&self, path: &str) -> Option<String> {
1518        if let Some(h) = self.hashes.borrow().get(path) {
1519            return h.clone();
1520        }
1521        let h = hash_file(Path::new(path));
1522        self.hashes.borrow_mut().insert(path.to_string(), h.clone());
1523        h
1524    }
1525
1526    /// Whether one name resolves in one directory, answered once per run. The project root,
1527    /// `src/` and `types/` are probed by every module, and modules share the names they
1528    /// require, so this overlaps even more than the file hashes do.
1529    fn resolves(&self, dir: &str, name: &str) -> bool {
1530        let k = (dir.to_string(), name.to_string());
1531        if let Some(v) = self.dirs.borrow().get(&k) {
1532            return *v;
1533        }
1534        let v = resolves_in(Path::new(dir), name);
1535        self.dirs.borrow_mut().insert(k, v);
1536        v
1537    }
1538
1539    /// What `dir` offers for `names`, hashed.
1540    fn probe_hash(&self, dir: &str, names: &[String]) -> String {
1541        let mut h = blake3::Hasher::new();
1542        for n in names {
1543            h.update(n.as_bytes());
1544            h.update(if self.resolves(dir, n) {
1545                b"\x01"
1546            } else {
1547                b"\x00"
1548            });
1549        }
1550        h.finalize().to_hex().to_string()
1551    }
1552
1553    fn entry_path(&self, key: &Key) -> PathBuf {
1554        self.dir.join(format!("{}.json", key.hash))
1555    }
1556
1557    /// The grain this store was opened with. A caller that decides between a per-module
1558    /// walk and a whole-run lookup asks the store rather than re-reading the flags, so the
1559    /// two cannot disagree.
1560    pub fn mode(&self) -> Mode {
1561        self.opts.mode
1562    }
1563
1564    /// A truncated or hand-edited entry is a miss, not a crash.
1565    fn parse<T: serde::de::DeserializeOwned>(&self, raw: &str) -> Option<T> {
1566        match serde_json::from_str(raw) {
1567            Ok(e) => Some(e),
1568            Err(e) => {
1569                self.miss(&format!("unreadable entry: {e}"));
1570                None
1571            }
1572        }
1573    }
1574
1575    /// Every question an entry's probe asks again: each name its modules required, asked
1576    /// from `subject` (the module the entry is about; `None` for a whole-run entry), and
1577    /// each name a member of their require closures required, asked from that member
1578    /// ([`Module::closure_requires`]). Sorted and deduplicated.
1579    ///
1580    /// Both sides of a probe comparison have to derive this the same way, which is why it is
1581    /// one function rather than two loops.
1582    fn questions(modules: &[Module], subject: Option<&str>) -> Vec<Question> {
1583        let mut qs: Vec<Question> = modules
1584            .iter()
1585            .flat_map(|m| {
1586                let own = m
1587                    .requires
1588                    .iter()
1589                    .map(|r| (subject.map(str::to_string), r.module.clone()));
1590                let below = m
1591                    .closure_requires
1592                    .iter()
1593                    .map(|e| (Some(e.from.clone()), e.module.clone()));
1594                own.chain(below)
1595            })
1596            .collect();
1597        qs.sort();
1598        qs.dedup();
1599        qs
1600    }
1601
1602    /// The names the questions ask, whoever asks them, sorted and deduplicated: what a
1603    /// directory probe hashes, since a directory offers a name or does not whoever asks.
1604    fn names_of(questions: &[Question]) -> Vec<String> {
1605        let mut names: Vec<String> = questions.iter().map(|(_, n)| n.clone()).collect();
1606        names.sort();
1607        names.dedup();
1608        names
1609    }
1610
1611    /// Whether an entry still describes the world: written by this build, every file it read
1612    /// unchanged, and every directory it could have resolved in still answering the same way
1613    /// for the names it asked about.
1614    fn still_valid(
1615        &self,
1616        kind: &str,
1617        stamp: &Stamp,
1618        inputs: &[Input],
1619        probes: &[Probe],
1620        questions: &[Question],
1621    ) -> bool {
1622        let Some(current) = Stamp::for_kind(kind) else {
1623            return false;
1624        };
1625        if *stamp != current {
1626            self.miss("written by a different build");
1627            return false;
1628        }
1629        for i in inputs {
1630            match self.hash_of(&i.path) {
1631                Some(h) if h == i.hash => {}
1632                Some(_) => {
1633                    self.miss(&format!("changed: {}", i.path));
1634                    return false;
1635                }
1636                None => {
1637                    self.miss(&format!("gone: {}", i.path));
1638                    return false;
1639                }
1640            }
1641        }
1642        for p in probes {
1643            if p.dir == MODEL_PROBE {
1644                let Some(answers) = &self.answers else {
1645                    self.miss("recorded against a project model this run does not have");
1646                    return false;
1647                };
1648                if self.model_hash(answers, questions) != p.names {
1649                    self.miss(
1650                        "what a name this module or its require closure requires resolves to changed",
1651                    );
1652                    return false;
1653                }
1654                continue;
1655            }
1656            if self.answers.is_some() {
1657                self.miss("recorded without the project model this run has");
1658                return false;
1659            }
1660            if self.probe_hash(&p.dir, &Self::names_of(questions)) != p.names {
1661                self.miss(&format!(
1662                    "what {} offers for this module's requires changed",
1663                    p.dir
1664                ));
1665                return false;
1666            }
1667        }
1668        self.stats.borrow_mut().hits += 1;
1669        true
1670    }
1671
1672    /// What each file in the walk reported last time; `None` where it has to be checked.
1673    ///
1674    /// Under `WholeRun` this is all-or-nothing by construction: one entry covers the walk,
1675    /// so a single changed input means every module is checked.
1676    pub fn lookup_all(&self, keys: &[Key], run: &Key, files: usize) -> Vec<Option<Module>> {
1677        match self.opts.mode {
1678            Mode::PerModule => keys.iter().map(|k| self.lookup(k)).collect(),
1679            Mode::WholeRun => match self.lookup_run(run, files) {
1680                Some(ms) => ms.into_iter().map(Some).collect(),
1681                None => vec![None; files],
1682            },
1683        }
1684    }
1685
1686    /// What the module reported last time, if every input and probe still matches.
1687    pub fn lookup(&self, key: &Key) -> Option<Module> {
1688        let raw = std::fs::read_to_string(self.entry_path(key)).ok()?;
1689        let entry: Entry = self.parse(&raw)?;
1690        let questions = Self::questions(std::slice::from_ref(&entry.module), Some(&entry.subject));
1691        if !self.still_valid(
1692            &entry.kind,
1693            &entry.stamp,
1694            &entry.inputs,
1695            &entry.probes,
1696            &questions,
1697        ) {
1698            return None;
1699        }
1700        self.touch(key);
1701        Some(entry.module)
1702    }
1703
1704    /// Mark an entry as used, so that the sweep drops what is stale rather than what is
1705    /// merely old.
1706    ///
1707    /// Without this the shape you run most often is the one whose entries were written
1708    /// first, so it ages out while a lint flag you tried once survives — measured on a
1709    /// 58-module project, running four other shapes left the plain one replaying 24 of its
1710    /// 58 modules. Failing to touch costs a later re-check and nothing else, so every error
1711    /// here is ignored.
1712    fn touch(&self, key: &Key) {
1713        if let Ok(f) = std::fs::File::options()
1714            .write(true)
1715            .open(self.entry_path(key))
1716        {
1717            let _ = f.set_modified(std::time::SystemTime::now());
1718        }
1719    }
1720
1721    fn lookup_run(&self, key: &Key, files: usize) -> Option<Vec<Module>> {
1722        let raw = std::fs::read_to_string(self.entry_path(key)).ok()?;
1723        let entry: RunEntry = self.parse(&raw)?;
1724        if entry.modules.len() != files {
1725            self.miss("the walk visits a different number of files");
1726            return None;
1727        }
1728        let questions = Self::questions(&entry.modules, None);
1729        if !self.still_valid(RUN, &entry.stamp, &entry.inputs, &entry.probes, &questions) {
1730            return None;
1731        }
1732        self.touch(key);
1733        Some(entry.modules)
1734    }
1735
1736    /// Hash every path an entry has to compare next time, once each: a dependency reached
1737    /// from two modules is compared once. `None` if anything is unreadable, which drops the
1738    /// entry rather than storing one that would hit wrongly.
1739    fn inputs_for(&self, paths: impl Iterator<Item = String>) -> Option<Vec<Input>> {
1740        let mut paths: Vec<String> = paths.collect();
1741        paths.sort();
1742        paths.dedup();
1743        let mut inputs = Vec::with_capacity(paths.len());
1744        for p in paths {
1745            inputs.push(Input {
1746                path: p.clone(),
1747                hash: self.hash_of(&p)?,
1748            });
1749        }
1750        Some(inputs)
1751    }
1752
1753    /// The probes an entry records for `questions`: the model's answers, when this store has
1754    /// the model; otherwise one per directory — `dirs`, the caller's for the module, and the
1755    /// directory of every member of the closure that asked something, which is where a file
1756    /// outside any project resolves its own `require`s ([`search_dirs`]).
1757    fn probes_for(&self, dirs: &[PathBuf], questions: &[Question]) -> Vec<Probe> {
1758        if let Some(answers) = &self.answers {
1759            return vec![Probe {
1760                dir: MODEL_PROBE.to_string(),
1761                names: self.model_hash(answers, questions),
1762            }];
1763        }
1764        let askers = questions
1765            .iter()
1766            .filter_map(|(asker, _)| asker.as_deref())
1767            .flat_map(|a| search_dirs(Path::new(a)));
1768        let mut dirs: Vec<String> = dirs
1769            .iter()
1770            .cloned()
1771            .chain(askers)
1772            .map(|p| normal(&p))
1773            .collect();
1774        dirs.sort();
1775        dirs.dedup();
1776        let names = Self::names_of(questions);
1777        dirs.into_iter()
1778            .map(|d| {
1779                let h = self.probe_hash(&d, &names);
1780                Probe { dir: d, names: h }
1781            })
1782            .collect()
1783    }
1784
1785    /// Record what one module reported. Best-effort: a store that cannot be written leaves
1786    /// the next run to do the work again, which is slow rather than wrong.
1787    pub fn store_module(
1788        &self,
1789        key: &Key,
1790        file: &Path,
1791        extra_inputs: &[PathBuf],
1792        dirs: &[PathBuf],
1793        module: &Module,
1794    ) {
1795        let Some(stamp) = Stamp::for_kind(key.kind) else {
1796            return;
1797        };
1798        let paths = std::iter::once(normal(file))
1799            .chain(module.deps.iter().cloned())
1800            .chain(extra_inputs.iter().map(|p| normal(p)));
1801        let Some(inputs) = self.inputs_for(paths) else {
1802            return;
1803        };
1804        let subject = normal(file);
1805        let questions = Self::questions(std::slice::from_ref(module), Some(&subject));
1806        let entry = Entry {
1807            stamp,
1808            subject: subject.clone(),
1809            kind: key.kind.to_string(),
1810            inputs,
1811            probes: self.probes_for(dirs, &questions),
1812            module: module.clone(),
1813        };
1814        match self.write(key, &entry) {
1815            Ok(()) => self.stats.borrow_mut().stored += 1,
1816            Err(e) if self.opts.explain => eprintln!("htl cache: not stored ({e})"),
1817            Err(_) => {}
1818        }
1819    }
1820
1821    /// Record the whole walk as one entry. Its inputs are the union of every module's, so
1822    /// any edit anywhere misses — which is the behaviour this mode is chosen for.
1823    pub fn store_run(
1824        &self,
1825        key: &Key,
1826        files: &[PathBuf],
1827        extra_inputs: &[PathBuf],
1828        dirs: &[PathBuf],
1829        modules: &[Module],
1830    ) {
1831        let Some(stamp) = Stamp::current() else {
1832            return;
1833        };
1834        let paths = files
1835            .iter()
1836            .map(|f| normal(f))
1837            .chain(modules.iter().flat_map(|m| m.deps.iter().cloned()))
1838            .chain(extra_inputs.iter().map(|p| normal(p)));
1839        let Some(inputs) = self.inputs_for(paths) else {
1840            return;
1841        };
1842        let questions = Self::questions(modules, None);
1843        let entry = RunEntry {
1844            stamp,
1845            subjects: files.iter().map(|f| normal(f)).collect(),
1846            kind: key.kind.to_string(),
1847            inputs,
1848            probes: self.probes_for(dirs, &questions),
1849            modules: modules.to_vec(),
1850        };
1851        match self.write(key, &entry) {
1852            Ok(()) => self.stats.borrow_mut().stored += 1,
1853            Err(e) if self.opts.explain => eprintln!("htl cache: not stored ({e})"),
1854            Err(_) => {}
1855        }
1856    }
1857
1858    /// What `lshape-version` reported last time, if the declaration, the runtime, the
1859    /// project model's answer for `lshape` and any `package.path` directories it consulted
1860    /// still match.
1861    ///
1862    /// Does not touch [`Stats`]: those count the walk's modules.
1863    pub fn lookup_lshape(&self, key: &Key) -> Option<Vec<crate::Diagnostic>> {
1864        let raw = std::fs::read_to_string(self.entry_path(key)).ok()?;
1865        let entry: LshapeEntry = serde_json::from_str(&raw).ok()?;
1866        if entry.kind != LSHAPE || entry.stamp != Stamp::for_kind(LSHAPE)? {
1867            return None;
1868        }
1869        for i in &entry.inputs {
1870            if self.hash_of(&i.path).as_deref() != Some(i.hash.as_str()) {
1871                return None;
1872            }
1873        }
1874        let questions = vec![(None, "lshape".to_string())];
1875        let names = Self::names_of(&questions);
1876        for p in &entry.probes {
1877            if p.dir == MODEL_PROBE {
1878                let answers = self.answers.as_ref()?;
1879                if self.model_hash(answers, &questions) != p.names {
1880                    return None;
1881                }
1882            } else if self.probe_hash(&p.dir, &names) != p.names {
1883                return None;
1884            }
1885        }
1886        let mut out = Vec::with_capacity(entry.diagnostics.len());
1887        for r in &entry.diagnostics {
1888            let severity = crate::Severity::parse(&r.severity)?;
1889            let fix = r.fix.as_ref().map(FixJson::to_fix);
1890            out.push(r.item.to_diagnostic(severity, fix));
1891        }
1892        self.touch(key);
1893        Some(out)
1894    }
1895
1896    /// Record what `lshape-version` reported for `files` (the declaration and, when found,
1897    /// the runtime) and `dirs` (the search path directories `package.path` consulted when
1898    /// the model held a declaration alone).
1899    pub fn store_lshape(
1900        &self,
1901        key: &Key,
1902        files: &[PathBuf],
1903        dirs: &[PathBuf],
1904        diagnostics: &[crate::Diagnostic],
1905    ) {
1906        if files.is_empty() {
1907            return;
1908        }
1909        let Some(stamp) = Stamp::for_kind(key.kind) else {
1910            return;
1911        };
1912        let Some(inputs) = self.inputs_for(files.iter().map(|f| normal(f))) else {
1913            return;
1914        };
1915        let questions = vec![(None, "lshape".to_string())];
1916        let names = Self::names_of(&questions);
1917        let mut probes = Vec::new();
1918        if let Some(answers) = &self.answers {
1919            probes.push(Probe {
1920                dir: MODEL_PROBE.to_string(),
1921                names: self.model_hash(answers, &questions),
1922            });
1923        }
1924        let mut norm_dirs: Vec<String> = dirs.iter().map(|d| normal(d)).collect();
1925        norm_dirs.sort();
1926        norm_dirs.dedup();
1927        for d in norm_dirs {
1928            let h = self.probe_hash(&d, &names);
1929            probes.push(Probe { dir: d, names: h });
1930        }
1931        let recorded = diagnostics
1932            .iter()
1933            .map(|d| Recorded {
1934                severity: d.severity.as_str().to_string(),
1935                item: ItemJson::from_diagnostic(d),
1936                fix: d.fix.as_ref().map(FixJson::from_fix),
1937                dependency: None,
1938            })
1939            .collect();
1940        let entry = LshapeEntry {
1941            stamp,
1942            subjects: files.iter().map(|f| normal(f)).collect(),
1943            kind: key.kind.to_string(),
1944            inputs,
1945            probes,
1946            diagnostics: recorded,
1947        };
1948        let _ = self.write(key, &entry);
1949    }
1950
1951    /// Drop entries this run did not use, once the store has outgrown what the project
1952    /// warrants.
1953    ///
1954    /// Nothing else removes an entry. The key covers the paths as written, the lint
1955    /// selection and the working directory, so every distinct way of invoking the check
1956    /// leaves a full set of module entries behind — a lint flag tried once doubles the store
1957    /// permanently, and a deleted module's entry is never read again. `htl check` sees the
1958    /// whole graph in one process, which is what lets this be exact about the orphans rather
1959    /// than sampling the way ccache has to.
1960    ///
1961    /// Two passes. Entries whose recorded inputs have all gone describe modules that no
1962    /// longer exist, and go first. If the store is still over, the oldest go until it fits.
1963    ///
1964    /// **This uses mtimes, and #3's rule against them still holds.** That rule is about
1965    /// invalidation, where trusting a timestamp means replaying a stale result and reporting
1966    /// something untrue. Dropping an entry that was still good costs the check it would have
1967    /// skipped and nothing else. The two questions deserve different tools.
1968    ///
1969    /// "Oldest" is least recently *used*: a hit touches its entry, which is what makes the
1970    /// two differ.
1971    /// Without that it would mean least recently written, and the shape run most often —
1972    /// written first — would age out while a shape tried once survived.
1973    pub fn sweep(&self, keep: &[Key], files: usize) {
1974        let bound = self
1975            .opts
1976            .max_entries
1977            .unwrap_or_else(|| default_bound(files));
1978        let Ok(rd) = std::fs::read_dir(&self.dir) else {
1979            return;
1980        };
1981        let keep: std::collections::HashSet<&str> = keep.iter().map(|k| k.hash.as_str()).collect();
1982
1983        let mut all = 0usize;
1984        let mut candidates: Vec<(PathBuf, std::time::SystemTime)> = Vec::new();
1985        for e in rd.flatten() {
1986            let path = e.path();
1987            if path.extension().is_none_or(|x| x != "json") {
1988                continue;
1989            }
1990            all += 1;
1991            let stem = path
1992                .file_stem()
1993                .map(|s| s.to_string_lossy().into_owned())
1994                .unwrap_or_default();
1995            if keep.contains(stem.as_str()) {
1996                continue;
1997            }
1998            let mtime = e
1999                .metadata()
2000                .and_then(|m| m.modified())
2001                .unwrap_or(std::time::UNIX_EPOCH);
2002            candidates.push((path, mtime));
2003        }
2004        if all <= bound {
2005            return;
2006        }
2007
2008        let mut removed = 0usize;
2009        // Orphans first: an entry none of whose inputs still exist cannot be read again.
2010        candidates.retain(|(p, _)| {
2011            if all - removed <= bound || !self.is_orphan(p) {
2012                return true;
2013            }
2014            if std::fs::remove_file(p).is_ok() {
2015                removed += 1;
2016            }
2017            false
2018        });
2019        // Then by age, oldest first, until the store fits.
2020        if all - removed > bound {
2021            candidates.sort_by_key(|(_, t)| *t);
2022            for (p, _) in &candidates {
2023                if all - removed <= bound {
2024                    break;
2025                }
2026                if std::fs::remove_file(p).is_ok() {
2027                    removed += 1;
2028                }
2029            }
2030        }
2031        self.stats.borrow_mut().evicted += removed;
2032        if removed > 0 && self.opts.explain {
2033            eprintln!("htl cache: dropped {removed} of {all} entries (bound {bound})");
2034        }
2035    }
2036
2037    /// Whether every file an entry recorded as an input has gone. Unreadable entries count as
2038    /// orphans: nothing can use them either.
2039    fn is_orphan(&self, path: &Path) -> bool {
2040        let Ok(raw) = std::fs::read_to_string(path) else {
2041            return true;
2042        };
2043        let Ok(v) = serde_json::from_str::<serde_json::Value>(&raw) else {
2044            return true;
2045        };
2046        let Some(inputs) = v.get("inputs").and_then(|i| i.as_array()) else {
2047            return true;
2048        };
2049        !inputs
2050            .iter()
2051            .filter_map(|i| i.get("path").and_then(|p| p.as_str()))
2052            .any(|p| Path::new(p).exists())
2053    }
2054
2055    /// Whole or not at all: a temporary file in the same directory, then a rename. On every
2056    /// filesystem htl runs on that rename is atomic, so a concurrent reader sees either the
2057    /// previous entry or this one, never half of this one. ccache relies on exactly this
2058    /// and takes no locks at all.
2059    fn write<T: Serialize>(&self, key: &Key, entry: &T) -> Result<()> {
2060        std::fs::create_dir_all(&self.dir)?;
2061        let final_path = self.entry_path(key);
2062        let tmp = self
2063            .dir
2064            .join(format!(".{}.{}.tmp", key.hash, std::process::id()));
2065        std::fs::write(&tmp, serde_json::to_vec(entry)?)?;
2066        if let Err(e) = std::fs::rename(&tmp, &final_path) {
2067            let _ = std::fs::remove_file(&tmp);
2068            return Err(e.into());
2069        }
2070        Ok(())
2071    }
2072}
2073
2074#[cfg(test)]
2075mod tests {
2076    use super::*;
2077    use crate::core_common;
2078
2079    fn tempdir(name: &str) -> core_common::TempDir {
2080        let dir = core_common::tempdir("htl-core-cache", name);
2081        std::fs::create_dir_all(dir.join("src")).unwrap();
2082        dir
2083    }
2084
2085    /// A `module` entry is stamped without the binary, so one written by `htl` is served
2086    /// inside a proc macro (whose binary is `rustc`) and the other way round. A test-file
2087    /// entry keeps the binary in its stamp, as it always did.
2088    #[test]
2089    fn a_module_entry_is_served_whatever_binary_wrote_it() {
2090        let root = tempdir("portable");
2091        let file = root.join("src/m.tl");
2092        std::fs::write(&file, "return {}\n").unwrap();
2093        let cache = Cache::open(&root, Options::default()).unwrap();
2094        // What another binary writes for a module: the portable stamp. This binary's own
2095        // stamp is a different value, which is the point.
2096        assert_ne!(Stamp::current().unwrap(), Stamp::portable());
2097        let module = Module {
2098            diagnostics: Vec::new(),
2099            errors: 0,
2100            warnings: 0,
2101            lints: 0,
2102            deps: Vec::new(),
2103            requires: Vec::new(),
2104            global_sites: Vec::new(),
2105            markers: Vec::new(),
2106            record_sites: Vec::new(),
2107            closure_requires: Vec::new(),
2108            code: Some("return {}".into()),
2109            check: Some(CheckInfoJson::default()),
2110        };
2111        let entry = |kind: &str| Entry {
2112            stamp: Stamp::portable(),
2113            subject: normal(&file),
2114            kind: kind.to_string(),
2115            inputs: cache.inputs_for(std::iter::once(normal(&file))).unwrap(),
2116            probes: Vec::new(),
2117            module: module.clone(),
2118        };
2119        let mk = module_gen_key(&file, None);
2120        cache.write(&mk, &entry(MODULE)).unwrap();
2121        assert!(
2122            cache.lookup(&mk).is_some(),
2123            "a module entry: the binary is not part of its stamp"
2124        );
2125        let gk = gen_key(&file, None);
2126        cache.write(&gk, &entry(GEN)).unwrap();
2127        assert!(
2128            cache.lookup(&gk).is_none(),
2129            "a test-file entry from another binary: written by a different build"
2130        );
2131        // And what this binary stores under a module key is what any other reads.
2132        cache.store_module(&mk, &file, &[], &[], &module);
2133        let raw = std::fs::read_to_string(cache.entry_path(&mk)).unwrap();
2134        let stored: Entry = serde_json::from_str(&raw).unwrap();
2135        assert_eq!(stored.stamp, Stamp::portable());
2136    }
2137
2138    #[test]
2139    fn source_mentions_require_is_word_bounded() {
2140        assert!(source_mentions_require("local a = require(\"a\")\n"));
2141        assert!(source_mentions_require("require 'a'"));
2142        assert!(!source_mentions_require("local required_by = 1\n"));
2143        assert!(!source_mentions_require("local x = prerequire\n"));
2144        assert!(!source_mentions_require("local record c\nend\nreturn c\n"));
2145        assert!(!source_mentions_require(""));
2146    }
2147
2148    #[test]
2149    fn scratch_roots_are_recognised() {
2150        assert_eq!(
2151            scratch_root(Path::new("/w/target/package/host-0.1.0")),
2152            Some("under a `target` directory")
2153        );
2154        assert_eq!(scratch_root(Path::new("/w/host")), None);
2155        let cargo_home = std::env::var_os("CARGO_HOME")
2156            .map(PathBuf::from)
2157            .or_else(|| std::env::var_os("HOME").map(|h| PathBuf::from(h).join(".cargo")))
2158            .expect("CARGO_HOME or HOME");
2159        assert_eq!(
2160            scratch_root(&cargo_home.join("registry/src/index/htl-0.3.0")),
2161            Some("a registry checkout")
2162        );
2163    }
2164}