htl_core/cache.rs
1//! An on-disk cache of what checking one module reported (issues #3, #19).
2//!
3//! Building a checker costs about 13.5 ms and type-checking a few thousand lines of Teal
4//! costs about a second (`crates/htl-core/benches/check.rs`). A run that changed nothing
5//! should pay neither, and a run that changed one module should pay for that module and
6//! what depends on it rather than for the project.
7//!
8//! # One entry per module
9//!
10//! #3 shipped this as one entry per invocation, because at the time it looked as though
11//! skipping individual modules would change what the rest of them saw. Two things settled
12//! that: #21 made a file's result independent of its position in the walk, and measurement
13//! showed the shared module store to be an optimisation rather than a precondition — a
14//! dependency being checked first does not change what its requirer reports. So an entry
15//! is now per module, and a run is the sum of them.
16//!
17//! # What makes an entry
18//!
19//! The **key** is the module as this invocation names it — the path spelling the checker
20//! will put into the diagnostics, plus the lint selection and the working directory.
21//! `--strict` and `--format` are deliberately absent: they change how a run is summarized
22//! and what it exits with, not what any module reports, so runs that differ only in those
23//! share their modules' entries. A level is in the key because it is written in the same spec as
24//! which rules run — moving one rule between `warn` and `deny` changes no diagnostic, and
25//! re-checks anyway.
26//!
27//! The **inputs** are the module and its require closure, by content hash: every file the
28//! check read through `require`, however many requires away ([`CheckInfo::deps`]). The
29//! closure and not the direct requires, because a check reads further than the module's
30//! own `require` calls. A `global` reaches a file through the whole chain (#370), and a
31//! record a required module re-exports (`type Std = h.Std`) is the record of the file it
32//! came from: `top.tl` requiring `mid.tl` requiring `host.d.tl` is typed by `host.d.tl`
33//! without ever naming it. An entry that hashed only `mid.tl` replayed `top.tl`'s old
34//! answer over an edit to `host.d.tl` — no error where the edit put two, and a
35//! `global-redeclaration` lint from the line numbers `top.tl` had seen before the edit
36//! beside the ones `mid.tl` saw after (#400). The cost is that an edit to a leaf re-checks
37//! everything above it, which is the right answer: the leaf's types reach all of it.
38//! Hashing is microseconds and memoised per run, so a project whose every module sits on
39//! one `.d.tl` hashes that file once.
40//!
41//! The **probes** say what each name the closure required resolves to — the module's own
42//! requires, asked from the module, and every member's, asked from that member
43//! ([`CheckInfo::closure_requires`]). A new `.tl` appearing somewhere changes that while
44//! every recorded hash still matches, and nothing else would catch it. This is the hole
45//! ccache documents in its direct mode, and it is as deep as the closure: a file appearing
46//! under a name `mid.tl` requires changes what `top.tl` sees exactly as an edit does. With
47//! the project's model ([`Cache::with_answers`]) a probe is the model's answer for every
48//! such question, each asked from the file that asked it: one hash, and the questions it
49//! hashes are the ones that matter. Without one (a file outside any project) a probe is a
50//! directory the search path consults — the module's own, and each member's that asked
51//! something — by the set of those names it offers.
52//!
53//! # No mtimes anywhere
54//!
55//! Content hashes only. Timestamp-based invalidation is where this class of tool
56//! historically breaks — second-granular filesystems on macOS, mtimes zeroed by Docker
57//! layers, clock skew, fresh CI checkouts invalidating everything — and hashing a module
58//! costs microseconds against the milliseconds it saves.
59//!
60//! # Failure is a miss
61//!
62//! Every error path here returns "no entry" rather than propagating. A corrupt file, an
63//! unreadable directory, a store on a read-only filesystem: the module gets checked, as it
64//! would have been anyway. The one invariant worth stating is mypy's: an entry is written
65//! whole or not at all, via a temporary file and a rename, so a reader never sees half of
66//! one. Set `HTL_CACHE_DEBUG=1` to print why a lookup missed.
67//!
68//! # Why this lives in htl-core
69//!
70//! The store began in the CLI, which was the only reader. The linker is a reader too
71//! (#100): `htl build` and `include_bundle!` walk the require closure through
72//! [`crate::link`], and a module whose `gen` entry still holds — the same entry `htl test`
73//! writes and replays — has no reason to be generated again. `htl-cli` depends on `htl`,
74//! which depends on `htl-macros`, so a store the macros can open has to sit below both.
75//! The CLI keeps its flags, its `cache status` command and its report types; what moved is
76//! the store and the JSON shapes it writes, and the conversions between those shapes and
77//! [`crate::CheckInfo`], which every reader was carrying a copy of.
78//!
79//! Only `htl check` sweeps ([`Cache::sweep`]). It is the one reader that sees the whole
80//! project, so its keep-list is the whole project's; a build or a macro expansion sees one
81//! closure, and sweeping from that view would evict every other entry the project has.
82
83use anyhow::Result;
84use serde::{Deserialize, Serialize};
85use std::cell::RefCell;
86use std::collections::HashMap;
87use std::path::{Path, PathBuf};
88
89use crate::{CheckInfo, DependencyError, Fix, RequireSite};
90
91/// Bumped by hand when anything below changes shape. An entry stamped with a different
92/// value is a miss rather than an error: a fresh checkout and an upgrade both take that
93/// path in normal operation, which is why rustc treats its own header mismatch the same
94/// way.
95///
96/// 6: the stamp carries the checker's identity, and a `module` entry carries nothing
97/// else about the binary (#100), so the proc macros — whose `current_exe` is rustc, which
98/// does not change when htl does — still miss on a checker that no longer exists, and
99/// the CLI and the macros read each other's module entries.
100///
101/// 11: an entry's inputs are the module's require closure, and a [`Module`] carries the
102/// names each member of the closure required ([`Module::closure_requires`]), which the
103/// probe asks again (#400). An entry of 10 hashed only the direct requires and would hit
104/// over an edit two requires away — exactly the entries this change exists to stop
105/// replaying — so they are a miss rather than a default.
106///
107/// 12: an entry now carries the module's marker census ([`Module::markers`],
108/// [`CheckInfoJson::markers`]) — `htl adopt`'s input (#304). An entry of 11 has no such
109/// field, and a replay of one would give `htl adopt` nothing for a file the run did not
110/// check.
111///
112/// 13: an entry now also carries the module's construction-site census
113/// ([`Module::record_sites`], [`CheckInfoJson::record_sites`]) — `unmarked-struct`'s input
114/// and `htl adopt`'s `---@struct` row (#304). An entry of 12 has no such field, and
115/// `unmarked_structs` reads every file of the walk, replayed ones included, the same
116/// reason `markers` and `global_sites` carry through a replay.
117///
118/// 14: the census of 13 widened from construction sites alone to every site of a
119/// declared record, `as` casts included, and each [`RecordSiteJson`] now carries `kind`
120/// (`"literal"` / `"cast"`) and `sealed` beside what it already had — `unmarked-sealed`'s
121/// input (#304), beside `unmarked-struct`'s. An entry of 13 has neither field, and
122/// `unmarked_sealeds` reads every file of the walk the same way `unmarked_structs` does,
123/// so a replay of one would give it nothing for a file the run did not check.
124///
125/// A change to the Lua a `module` entry carries needs no bump: the stamp's `checker` is a
126/// hash of `prelude.lua` among the rest ([`crate::checker_identity`]), so a generator that
127/// emits different text is a different checker and every warm entry misses on its own.
128/// What this number is for is a change to the shape of what is stored — a field, a key, a
129/// meaning — which the hash cannot see.
130const FORMAT: u32 = 14;
131
132/// Where the store lives under the project root. Generated, and `htl init` puts `.htl/` in
133/// `.gitignore` — one line for the cache and the installed deps beside it, both
134/// machine-local and regenerated; an existing project adds the line by hand.
135const DIR: &str = ".htl/cache";
136
137/// How many entries a project's store may hold before a run starts dropping the ones it did
138/// not use.
139///
140/// Scaled to the project, because the natural size is one entry per module per way of
141/// invoking the check, and "a few ways" is what people do. The floor keeps a small project
142/// from evicting itself on its second invocation.
143fn default_bound(files: usize) -> usize {
144 (files * 4).max(256)
145}
146
147/// What produced an entry. A cache is only as good as its ability to notice it was
148/// written by something else.
149///
150/// The binary's length and modification time stand in for "which checker is this": during
151/// development the vendored Teal compiler, the prelude and the Rust side all change many
152/// times within one released version number, and a stamp that only carried the version
153/// would happily replay results from a checker that no longer exists. Rebuilding always
154/// moves both. This is the same reasoning behind sccache hashing the compiler binary into
155/// its key.
156///
157/// A [`MODULE`] entry — a required module's generated Lua — is stamped without the
158/// binary. Those entries are written by `htl test`, `htl build` and the proc macros, and
159/// inside a proc macro the binary is `rustc`: stamping with it would have the CLI and the
160/// macros overwrite each other's entries for the same module forever. What decides what a
161/// module generates is the Lua the checker is made of (the vendored `tl`, the prelude, the
162/// lints), so those entries carry [`crate::checker_identity`] — a hash of exactly that —
163/// beside the format and the htl version, and nothing that differs between two binaries
164/// built from the same sources.
165#[derive(Serialize, Deserialize, PartialEq, Eq, Debug)]
166struct Stamp {
167 format: u32,
168 htl: String,
169 checker: String,
170 exe_len: u64,
171 exe_mtime_ns: u128,
172}
173
174impl Stamp {
175 /// What this build stamps an entry of `kind` with: the binary too, except for
176 /// [`MODULE`] entries (see the type doc).
177 fn for_kind(kind: &str) -> Option<Self> {
178 if kind == MODULE {
179 Some(Self::portable())
180 } else {
181 Self::current()
182 }
183 }
184
185 fn current() -> Option<Self> {
186 let exe = std::env::current_exe().ok()?;
187 let m = std::fs::metadata(&exe).ok()?;
188 let mtime = m
189 .modified()
190 .ok()?
191 .duration_since(std::time::UNIX_EPOCH)
192 .ok()?
193 .as_nanos();
194 Some(Self {
195 exe_len: m.len(),
196 exe_mtime_ns: mtime,
197 ..Self::portable()
198 })
199 }
200
201 /// The stamp with nothing about the binary in it.
202 fn portable() -> Self {
203 Self {
204 format: FORMAT,
205 htl: env!("CARGO_PKG_VERSION").to_string(),
206 checker: crate::checker_identity().to_string(),
207 exe_len: 0,
208 exe_mtime_ns: 0,
209 }
210 }
211}
212
213/// One file the check read, by content.
214#[derive(Serialize, Deserialize, Debug)]
215struct Input {
216 path: String,
217 hash: String,
218}
219
220/// One directory a `require` could have resolved in, and what it offered *for the names
221/// this module asked for*.
222///
223/// This catches a file appearing where the checker would now find it, which no hash of the
224/// files it did read can see — the hole ccache documents in its direct mode. It used to
225/// hash every module name in the directory, which meant adding any file at all invalidated
226/// every module whose search path included it: writing one new module re-checked the whole
227/// project (#24). A file appearing under a name nobody requires cannot change what anybody
228/// resolved, so only the requested names go in.
229#[derive(Serialize, Deserialize, Debug)]
230struct Probe {
231 dir: String,
232 /// Hash of `(name, whether it resolves here)` over the names the module and the
233 /// members of its require closure required, sorted. Changing which directory a name
234 /// resolves in changes this for both directories involved, which is how a shadowing
235 /// file is caught. For the [`MODEL_PROBE`], the hash of the model's answer to each
236 /// question — a name, asked from the file that required it ([`Cache::questions`]).
237 names: String,
238}
239
240/// One question a probe asks again: a name, and the file that required it (`None` for the
241/// requires of a whole-run entry's modules, asked from nowhere in particular, as before
242/// the closure was probed).
243type Question = (Option<String>, String);
244
245/// The `dir` of the one probe a store with the project model records: not a directory,
246/// the model's answers ([`Cache::with_answers`]).
247const MODEL_PROBE: &str = "(project model)";
248
249/// The project model's answer for a module name, asked from a file (`None` for a whole
250/// run), as a string that changes whenever the answer does — which module, which files.
251/// What a store with the model probes instead of directories ([`Cache::with_answers`]).
252pub type Answers = std::sync::Arc<dyn Fn(Option<&Path>, &str) -> String + Send + Sync>;
253
254/// One diagnostic exactly as it was handed to the sink, so a replay goes through the same
255/// printing code the original run did rather than through a reconstruction of it.
256/// Reconstructing text from parsed fields is how a cache starts printing subtly different
257/// output from the run it claims to reproduce.
258#[derive(Serialize, Deserialize, Debug, Clone)]
259pub struct Recorded {
260 /// [`Severity::as_str`](crate::Severity::as_str)'s word. A string rather than the enum
261 /// because an entry outlives the build that wrote it, and a word this build does not
262 /// know has to read back as itself rather than fail the whole entry.
263 pub severity: String,
264 /// The diagnostic's parts. Its text is made from them by the one formatter
265 /// ([`crate::Diagnostic`]'s `Display`) at replay as it was at the original run, which is
266 /// why storing the parts reproduces the run: there is no second way to spell the line.
267 pub item: ItemJson,
268 /// The fix the diagnostic carried, for `htl fix` replaying instead of re-checking.
269 /// `None` when the diagnostic had none, which is most of them.
270 pub fix: Option<FixJson>,
271 /// Set when the text is an error in a module this one required rather than in this
272 /// one: the dependency and the file that required it. The entry carries every such
273 /// error the check found; the sink decides at replay, as it did at the original run,
274 /// which of them to say (once per run). Absent in entries written before this field
275 /// existed, which then read as having none.
276 #[serde(default, skip_serializing_if = "Option::is_none")]
277 pub dependency: Option<DependencyJson>,
278}
279
280/// A [`Fix`] as an entry (and the CLI's `--format json`) stores it.
281#[derive(Serialize, Deserialize, Debug, Clone)]
282pub struct FixJson {
283 /// `safe` / `unsafe` / `suggest`. Owned rather than `&'static str` because the store
284 /// reads these back, and a borrowed field cannot be deserialized into. The JSON is
285 /// unchanged either way.
286 pub applicability: String,
287 /// The edits, in the order [`Fix`] holds them. A fix is all of them or none: applying
288 /// part of one leaves the file in a state nobody asked for.
289 pub edits: Vec<EditJson>,
290}
291
292/// One replacement in a [`FixJson`]: a half-open span and what goes there.
293///
294/// Positions are the checker's — lines and columns counted from 1 — rather than byte
295/// offsets, because that is what the diagnostic beside it says and an entry that stored
296/// them differently would have to be trusted to convert the same way twice.
297#[derive(Serialize, Deserialize, Debug, Clone)]
298pub struct EditJson {
299 /// First line of the span.
300 pub line: usize,
301 /// First column of the span, inclusive.
302 pub col: usize,
303 /// Last line of the span; the same as `line` for an edit inside one line.
304 pub end_line: usize,
305 /// Column the span stops before, exclusive — so an empty span is an insertion.
306 pub end_col: usize,
307 /// What replaces the span. Empty to delete it.
308 pub text: String,
309}
310
311impl FixJson {
312 /// A fix as an entry stores it, for the run that writes one.
313 pub fn from_fix(f: &Fix) -> Self {
314 Self {
315 applicability: f.applicability.as_str().to_string(),
316 edits: f
317 .edits
318 .iter()
319 .map(|e| EditJson {
320 line: e.line,
321 col: e.col,
322 end_line: e.end_line,
323 end_col: e.end_col,
324 text: e.text.clone(),
325 })
326 .collect(),
327 }
328 }
329
330 /// And back, for a replayed one. An applicability this build does not know reads as
331 /// the most cautious of the three rather than failing the entry — see the match below.
332 pub fn to_fix(&self) -> Fix {
333 Fix {
334 applicability: match self.applicability.as_str() {
335 "unsafe" => crate::Applicability::Unsafe,
336 "suggest" => crate::Applicability::Suggest,
337 // Anything else is a build that wrote a name this one does not know;
338 // treating it as the most cautious of the three is the only safe reading.
339 "safe" => crate::Applicability::Safe,
340 _ => crate::Applicability::Suggest,
341 },
342 edits: self
343 .edits
344 .iter()
345 .map(|e| crate::Edit {
346 line: e.line,
347 col: e.col,
348 end_line: e.end_line,
349 end_col: e.end_col,
350 text: e.text.clone(),
351 })
352 .collect(),
353 }
354 }
355}
356
357/// What a dependency's diagnostic carries besides its text: the file that required it
358/// and where the file lives. Stored with the diagnostic ([`Recorded`]) so a replay says
359/// exactly what the run said, and decides the same way whether to say it.
360#[derive(Serialize, Deserialize, Debug, Clone)]
361pub struct DependencyJson {
362 /// The file the error is in, as the checker found it.
363 pub file: String,
364 /// The file whose `require` pulled it in — which is the one the reader is looking at,
365 /// and so the one a message about somebody else's file has to name.
366 pub required_by: String,
367 /// `dependency` / `external`, or none for a file of the project's own.
368 #[serde(default, skip_serializing_if = "Option::is_none")]
369 pub origin: Option<String>,
370}
371
372/// One literal `require` and where the checker resolved it. Kept because the project-level
373/// cycle lint runs over every file's requires, replayed ones included — a cycle that closes
374/// through a module nobody edited is still a cycle.
375#[derive(Serialize, Deserialize, Debug, Clone)]
376pub struct RequireJson {
377 /// The name as the source spells it.
378 pub module: String,
379 /// What it resolved to, `None` when the checker found nothing — a host module, or a
380 /// require that will fail. Stored either way: the cycle lint reads the resolved ones,
381 /// and a name that starts resolving is a change this entry has to notice.
382 pub path: Option<String>,
383 /// Where the `require` is, so a finding about it points at the call rather than the
384 /// file.
385 pub line: usize,
386 /// Column of the same.
387 pub col: usize,
388 /// [`RequireSite::erased`]: the value of a `local type` the generator drops. Stored so a
389 /// replayed module's requires say what the fresh check's would; absent in entries
390 /// written before the field existed, which is why the format was bumped rather than
391 /// defaulted — an old entry for an unchanged file would otherwise keep counting the
392 /// erased require as the host's.
393 #[serde(default)]
394 pub erased: bool,
395}
396
397/// What checking one module reported.
398#[derive(Serialize, Deserialize, Debug, Clone)]
399pub struct Module {
400 /// Its diagnostics, its contract lints included, in the order they were printed.
401 pub diagnostics: Vec<Recorded>,
402 /// How many of `diagnostics` were errors, counted when the entry was written.
403 ///
404 /// Stored rather than recounted on replay: the three are what a run adds up to decide
405 /// its exit code, and a replay that recounted them would be deciding that from its own
406 /// reading of the severity strings instead of from what the original run concluded.
407 pub errors: usize,
408 /// Warnings, as `errors`.
409 pub warnings: usize,
410 /// Lints, as `errors`.
411 pub lints: usize,
412 /// Every file the check read through `require`: the module's require closure
413 /// ([`CheckInfo::deps`]), not only its own requires. The next run keys on these, which
414 /// is how a dependency's edit invalidates its dependents — every one above it, however
415 /// many requires away.
416 pub deps: Vec<String>,
417 /// Every `require` in the source and where it went, which `deps` is the resolved,
418 /// deduplicated half of. Kept whole because the cycle lint reports a call site, and a
419 /// path cannot say which line asked for it.
420 pub requires: Vec<RequireJson>,
421 /// Every `global` declaration the check brought into scope and where
422 /// ([`CheckInfo::global_sites`]), kept for the same reason `requires` is: the
423 /// `global-redeclaration` lint runs over every file of the walk, replayed ones
424 /// included, and a declaration file is never a walked file, so a replayed module has
425 /// to carry the sites it saw. Absent in entries written before the field existed.
426 #[serde(default)]
427 pub global_sites: Vec<GlobalSiteJson>,
428 /// The marker census the check found on the file's declarations
429 /// ([`CheckInfo::markers`]), kept for the same reason `global_sites` is: `htl adopt`
430 /// (#304) reads every file of the walk, replayed ones included, and the module that
431 /// carries them is all a replay has. Absent in entries written before the field
432 /// existed.
433 #[serde(default)]
434 pub markers: Vec<MarkerSiteJson>,
435 /// The census of record sites the check found in the file
436 /// ([`CheckInfo::record_sites`]), kept for the same reason `markers` is: a run
437 /// replayed from the cache still feeds `unmarked_structs`, `unmarked_sealeds` and
438 /// `htl adopt`'s `---@struct` and `---@sealed` rows (#304), and the module that
439 /// carries them is all a replay has. Absent in entries written before the field
440 /// existed.
441 #[serde(default)]
442 pub record_sites: Vec<RecordSiteJson>,
443 /// The names each member of the require closure required — its require sites, resolved
444 /// or not ([`CheckInfo::closure_requires`]): the questions below the module that the
445 /// probe asks again, each from the member that asked it. `deps` says the files read are unchanged;
446 /// this is what says each name still means the file it meant. Entries written before the
447 /// field existed are a format behind (the stamp) and not read.
448 #[serde(default)]
449 pub closure_requires: Vec<ClosureRequireJson>,
450 /// The Lua this module generates, for entries under [`gen_key`]. `htl check` never needs
451 /// it and stores `None`; `htl test` stores it so a replay can go straight to running.
452 /// Absent when checking produced errors, since there is nothing to run then.
453 #[serde(default)]
454 pub code: Option<String>,
455 /// What checking reported, in the form the test runner needs it back.
456 ///
457 /// `htl check` replays `diagnostics` straight into the sink and never reconstructs a
458 /// `CheckInfo`; the test runner puts one into its report, so a replayed test file needs
459 /// the structured form rather than the printed one. Only `gen_key` entries carry it.
460 #[serde(default)]
461 pub check: Option<CheckInfoJson>,
462}
463
464/// A `CheckInfo` as an entry stores it.
465///
466/// Deliberately a separate type from the diagnostics `htl check` replays: those are text
467/// on their way to a terminal, these are the fields the runner reads back.
468#[derive(Serialize, Deserialize, Debug, Clone, Default)]
469pub struct CheckInfoJson {
470 /// [`CheckInfo::errors`], verbatim.
471 pub errors: Vec<String>,
472 /// [`CheckInfo::warnings`], verbatim.
473 pub warnings: Vec<String>,
474 /// [`CheckInfo::lints`], verbatim.
475 pub lints: Vec<String>,
476 /// [`CheckInfo::deps`] as strings. `PathBuf` does not round-trip through JSON on every
477 /// platform; these are normalised on the way in and rebuilt on the way out.
478 pub deps: Vec<String>,
479 /// [`CheckInfo::requires`], in this module's own JSON shape.
480 pub requires: Vec<RequireJson>,
481 /// Parallel to `errors`, as in [`CheckInfo::error_fixes`]: `error_fixes[i]` belongs to
482 /// `errors[i]`. Stored as a full-length vector of `Option` rather than as the fixes
483 /// that exist, so the pairing survives without an index.
484 pub error_fixes: Vec<Option<FixJson>>,
485 /// Parallel to `lints`, as `error_fixes` is to `errors`.
486 pub lint_fixes: Vec<Option<FixJson>>,
487 /// `CheckInfo::dependency_errors`, so the runner reads back the whole of what the
488 /// check said. Absent in entries written before the field existed.
489 #[serde(default)]
490 pub dependency_errors: Vec<DependencyErrorJson>,
491 /// [`CheckInfo::error_items`], [`CheckInfo::warning_items`] and
492 /// [`CheckInfo::lint_items`], in the same order as the text, the fixes left in
493 /// `error_fixes` / `lint_fixes` beside them. Entries written before these existed are
494 /// a format behind and are not read (the stamp's format).
495 #[serde(default)]
496 pub error_items: Vec<ItemJson>,
497 /// See `error_items`.
498 #[serde(default)]
499 pub warning_items: Vec<ItemJson>,
500 /// See `error_items`.
501 #[serde(default)]
502 pub lint_items: Vec<ItemJson>,
503 /// [`CheckInfo::syntax_errors`]. Absent in entries written before the field existed,
504 /// and read back as none — which they were: only a file that generated is stored with
505 /// its check, and a file the parser rejected generates nothing.
506 #[serde(default)]
507 pub syntax_errors: usize,
508 /// [`CheckInfo::global_sites`], so a run replayed from the cache still feeds the
509 /// `global-redeclaration` lint. Absent in entries written before the field existed;
510 /// those are a format behind (the stamp) and not read.
511 #[serde(default)]
512 pub global_sites: Vec<GlobalSiteJson>,
513 /// [`CheckInfo::markers`], so the test runner's report of a replayed module carries
514 /// the same census a fresh check would. Absent in entries written before the field
515 /// existed.
516 #[serde(default)]
517 pub markers: Vec<MarkerSiteJson>,
518 /// [`CheckInfo::record_sites`], so the test runner's report of a replayed module
519 /// carries the same census of record sites a fresh check would. Absent in entries
520 /// written before the field existed.
521 #[serde(default)]
522 pub record_sites: Vec<RecordSiteJson>,
523 /// [`CheckInfo::closure_requires`], so the check a replayed module hands back is the
524 /// whole of what the fresh one said.
525 #[serde(default)]
526 pub closure_requires: Vec<ClosureRequireJson>,
527}
528
529/// One edge below a module ([`CheckInfo::closure_requires`]) as an entry stores it: a member
530/// of the require closure and one name it required.
531#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
532pub struct ClosureRequireJson {
533 /// The member that required the name, as the checker found it.
534 pub from: String,
535 /// The name as that member's `require` spelt it.
536 pub module: String,
537}
538
539/// One [`crate::GlobalSite`] as an entry stores it.
540#[derive(Serialize, Deserialize, Debug, Clone)]
541pub struct GlobalSiteJson {
542 /// The declared name.
543 pub name: String,
544 /// The declaring file, as the checker found it.
545 pub file: String,
546 /// Line of the declared name.
547 pub line: usize,
548 /// Column of the same.
549 pub col: usize,
550}
551
552/// One [`crate::MarkerSite`] as an entry stores it.
553///
554/// `PartialEq` beyond what the neighbouring `*SiteJson` types derive: a replayed module's
555/// census is compared against a fresh check's (`crates/htl-core/tests/marker_census.rs`),
556/// which the other site types have had no reason to do yet.
557#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
558pub struct MarkerSiteJson {
559 /// The marker's name, without its `---@`.
560 pub marker: String,
561 /// What the marker is on: `"record"`, `"field"` or `"function"`.
562 pub kind: String,
563 /// Line of the declaration the marker is read from.
564 pub line: usize,
565 /// The declaration's name.
566 pub name: String,
567}
568
569/// One [`crate::RecordSite`] as an entry stores it.
570///
571/// `PartialEq`, for the same reason `MarkerSiteJson` carries it: a replayed module's
572/// census is compared against a fresh check's
573/// (`crates/htl-core/tests/record_sites.rs`).
574#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
575pub struct RecordSiteJson {
576 /// [`crate::RecordSite::record_file`].
577 pub record_file: String,
578 /// [`crate::RecordSite::record_line`].
579 pub record_line: usize,
580 /// [`crate::RecordSite::record_name`].
581 pub record_name: String,
582 /// [`crate::RecordSite::marked`].
583 pub marked: bool,
584 /// [`crate::RecordSite::line`].
585 pub line: usize,
586 /// [`crate::RecordSite::col`].
587 pub col: usize,
588 /// [`crate::RecordSite::kind`].
589 pub kind: String,
590 /// [`crate::RecordSite::sealed`].
591 pub sealed: bool,
592 /// [`crate::RecordSite::complete`].
593 pub complete: bool,
594}
595
596/// A [`crate::Diagnostic`]'s parts as an entry stores them: the position, the rule and the
597/// sentence. Its severity is the list it is in, and its fix is the one stored beside it.
598#[derive(Serialize, Deserialize, Debug, Clone, Default)]
599pub struct ItemJson {
600 /// [`Diagnostic::file`](crate::Diagnostic::file).
601 pub file: String,
602 /// [`Diagnostic::line`](crate::Diagnostic::line).
603 pub line: usize,
604 /// [`Diagnostic::col`](crate::Diagnostic::col).
605 pub col: usize,
606 /// [`Diagnostic::rule`](crate::Diagnostic::rule).
607 pub rule: Option<String>,
608 /// [`Diagnostic::message`](crate::Diagnostic::message).
609 pub message: String,
610}
611
612impl ItemJson {
613 /// `d`'s parts as an entry stores them.
614 pub fn from_diagnostic(d: &crate::Diagnostic) -> Self {
615 Self {
616 file: d.file.clone(),
617 line: d.line,
618 col: d.col,
619 rule: d.rule.clone(),
620 message: d.message.clone(),
621 }
622 }
623
624 /// The diagnostic these parts are, with `severity` and `fix`.
625 pub fn to_diagnostic(
626 &self,
627 severity: crate::Severity,
628 fix: Option<crate::Fix>,
629 ) -> crate::Diagnostic {
630 crate::Diagnostic {
631 severity,
632 file: self.file.clone(),
633 line: self.line,
634 col: self.col,
635 rule: self.rule.clone(),
636 message: self.message.clone(),
637 fix,
638 required_by: None,
639 origin: None,
640 }
641 }
642}
643
644impl CheckInfoJson {
645 /// What a check reported, in the form an entry stores it.
646 pub fn from_check(c: &CheckInfo) -> Self {
647 let items = |ds: &[crate::Diagnostic]| ds.iter().map(ItemJson::from_diagnostic).collect();
648 Self {
649 error_items: items(&c.error_items),
650 warning_items: items(&c.warning_items),
651 lint_items: items(&c.lint_items),
652 errors: c.errors.clone(),
653 syntax_errors: c.syntax_errors,
654 warnings: c.warnings.clone(),
655 lints: c.lints.clone(),
656 deps: c.deps.iter().map(|p| normal(p)).collect(),
657 requires: requires_json(c),
658 error_fixes: c
659 .error_fixes
660 .iter()
661 .map(|f| f.as_ref().map(FixJson::from_fix))
662 .collect(),
663 lint_fixes: c
664 .lint_fixes
665 .iter()
666 .map(|f| f.as_ref().map(FixJson::from_fix))
667 .collect(),
668 dependency_errors: c
669 .dependency_errors
670 .iter()
671 .map(|e| DependencyErrorJson {
672 file: e.file.display().to_string(),
673 required_by: e.required_by.display().to_string(),
674 text: e.text.clone(),
675 item: ItemJson::from_diagnostic(&e.diagnostic),
676 })
677 .collect(),
678 global_sites: global_sites_json(c),
679 markers: marker_sites_json(c),
680 record_sites: record_sites_json(c),
681 closure_requires: closure_requires_json(c),
682 }
683 }
684
685 /// And back, for a replayed module: the test runner puts it into its report, the
686 /// linker reads its lints, requires and dependency errors.
687 pub fn to_check(&self) -> CheckInfo {
688 let error_fixes: Vec<Option<crate::Fix>> = self
689 .error_fixes
690 .iter()
691 .map(|f| f.as_ref().map(FixJson::to_fix))
692 .collect();
693 let lint_fixes: Vec<Option<crate::Fix>> = self
694 .lint_fixes
695 .iter()
696 .map(|f| f.as_ref().map(FixJson::to_fix))
697 .collect();
698 let items = |list: &[ItemJson], severity, fixes: &[Option<crate::Fix>]| {
699 list.iter()
700 .enumerate()
701 .map(|(i, it)| it.to_diagnostic(severity, fixes.get(i).and_then(|f| f.clone())))
702 .collect()
703 };
704 CheckInfo {
705 error_items: items(&self.error_items, crate::Severity::Error, &error_fixes),
706 warning_items: items(&self.warning_items, crate::Severity::Warning, &[]),
707 lint_items: items(&self.lint_items, crate::Severity::Lint, &lint_fixes),
708 errors: self.errors.clone(),
709 syntax_errors: self.syntax_errors,
710 warnings: self.warnings.clone(),
711 lints: self.lints.clone(),
712 deps: self.deps.iter().map(PathBuf::from).collect(),
713 requires: requires_from_json(&self.requires),
714 error_fixes,
715 lint_fixes,
716 dependency_errors: self
717 .dependency_errors
718 .iter()
719 .map(|e| DependencyError {
720 file: PathBuf::from(&e.file),
721 required_by: PathBuf::from(&e.required_by),
722 text: e.text.clone(),
723 diagnostic: e.item.to_diagnostic(crate::Severity::Error, None),
724 })
725 .collect(),
726 global_sites: global_sites_from_json(&self.global_sites),
727 markers: marker_sites_from_json(&self.markers),
728 record_sites: record_sites_from_json(&self.record_sites),
729 closure_requires: closure_requires_from_json(&self.closure_requires),
730 types: Default::default(),
731 }
732 }
733}
734
735/// One `htl::DependencyError` as an entry stores it.
736#[derive(Serialize, Deserialize, Debug, Clone)]
737pub struct DependencyErrorJson {
738 /// The required module the error is in.
739 pub file: String,
740 /// The file whose `require` reached it — the one being checked, and the one a report
741 /// names.
742 pub required_by: String,
743 /// The error as the dependency's own check phrased it.
744 pub text: String,
745 /// The same error in its parts ([`crate::DependencyError::diagnostic`]).
746 #[serde(default)]
747 pub item: ItemJson,
748}
749
750/// `CheckInfo`'s requires in the form an entry stores them.
751pub fn requires_json(c: &CheckInfo) -> Vec<RequireJson> {
752 c.requires
753 .iter()
754 .map(|r| RequireJson {
755 module: r.module.clone(),
756 path: r.path.as_ref().map(|p| p.to_string_lossy().into_owned()),
757 line: r.line,
758 col: r.col,
759 erased: r.erased,
760 })
761 .collect()
762}
763
764/// [`CheckInfo::global_sites`] in the shape an entry stores.
765pub fn global_sites_json(c: &CheckInfo) -> Vec<GlobalSiteJson> {
766 c.global_sites
767 .iter()
768 .map(|s| GlobalSiteJson {
769 name: s.name.clone(),
770 file: s.file.display().to_string(),
771 line: s.line,
772 col: s.col,
773 })
774 .collect()
775}
776
777/// [`CheckInfo::markers`] in the shape an entry stores.
778pub fn marker_sites_json(c: &CheckInfo) -> Vec<MarkerSiteJson> {
779 c.markers
780 .iter()
781 .map(|s| MarkerSiteJson {
782 marker: s.marker.clone(),
783 kind: s.kind.clone(),
784 line: s.line,
785 name: s.name.clone(),
786 })
787 .collect()
788}
789
790/// [`CheckInfo::record_sites`] in the shape an entry stores.
791pub fn record_sites_json(c: &CheckInfo) -> Vec<RecordSiteJson> {
792 c.record_sites
793 .iter()
794 .map(|s| RecordSiteJson {
795 record_file: s.record_file.clone(),
796 record_line: s.record_line,
797 record_name: s.record_name.clone(),
798 marked: s.marked,
799 line: s.line,
800 col: s.col,
801 kind: s.kind.clone(),
802 sealed: s.sealed,
803 complete: s.complete,
804 })
805 .collect()
806}
807
808/// [`CheckInfo::closure_requires`] in the shape an entry stores.
809pub fn closure_requires_json(c: &CheckInfo) -> Vec<ClosureRequireJson> {
810 c.closure_requires
811 .iter()
812 .map(|(from, module)| ClosureRequireJson {
813 from: normal(from),
814 module: module.clone(),
815 })
816 .collect()
817}
818
819fn closure_requires_from_json(edges: &[ClosureRequireJson]) -> Vec<(PathBuf, String)> {
820 edges
821 .iter()
822 .map(|e| (PathBuf::from(&e.from), e.module.clone()))
823 .collect()
824}
825
826fn global_sites_from_json(sites: &[GlobalSiteJson]) -> Vec<crate::GlobalSite> {
827 sites
828 .iter()
829 .map(|s| crate::GlobalSite {
830 name: s.name.clone(),
831 file: PathBuf::from(&s.file),
832 line: s.line,
833 col: s.col,
834 })
835 .collect()
836}
837
838fn marker_sites_from_json(sites: &[MarkerSiteJson]) -> Vec<crate::MarkerSite> {
839 sites
840 .iter()
841 .map(|s| crate::MarkerSite {
842 marker: s.marker.clone(),
843 kind: s.kind.clone(),
844 line: s.line,
845 name: s.name.clone(),
846 })
847 .collect()
848}
849
850fn record_sites_from_json(sites: &[RecordSiteJson]) -> Vec<crate::RecordSite> {
851 sites
852 .iter()
853 .map(|s| crate::RecordSite {
854 record_file: s.record_file.clone(),
855 record_line: s.record_line,
856 record_name: s.record_name.clone(),
857 marked: s.marked,
858 line: s.line,
859 col: s.col,
860 kind: s.kind.clone(),
861 sealed: s.sealed,
862 complete: s.complete,
863 })
864 .collect()
865}
866
867fn requires_from_json(requires: &[RequireJson]) -> Vec<RequireSite> {
868 requires
869 .iter()
870 .map(|r| RequireSite {
871 module: r.module.clone(),
872 path: r.path.as_ref().map(PathBuf::from),
873 line: r.line,
874 col: r.col,
875 erased: r.erased,
876 })
877 .collect()
878}
879
880impl Module {
881 /// A `CheckInfo` carrying only what the project-level lints read.
882 ///
883 /// `require_cycles` runs over every file in the walk, replayed ones included — a cycle
884 /// that closes through a module nobody edited is still a cycle — so a replayed module
885 /// has to produce something that lint can read. `global_redeclarations` reads the
886 /// sites the same way, and the error texts, for the one error of the checker's it
887 /// defers to; the recorded diagnostics give those back. `unmarked_structs` and
888 /// `unmarked_sealeds` read `record_sites` the same way again: a record's
889 /// construction and cast sites are typically in files other than its own
890 /// declaration, so a project-level census that skipped a replayed module would
891 /// silently lose every site that module built or cast. Everything is already
892 /// printed by then, and nothing downstream looks at the other fields.
893 pub fn requires_only(&self) -> CheckInfo {
894 CheckInfo {
895 deps: self.deps.iter().map(PathBuf::from).collect(),
896 requires: requires_from_json(&self.requires),
897 global_sites: global_sites_from_json(&self.global_sites),
898 record_sites: record_sites_from_json(&self.record_sites),
899 closure_requires: closure_requires_from_json(&self.closure_requires),
900 errors: self
901 .diagnostics
902 .iter()
903 .filter(|r| r.severity == crate::Severity::Error.as_str())
904 .map(|r| r.item.message.clone())
905 .collect(),
906 ..Default::default()
907 }
908 }
909
910 /// A `gen` entry as the linker and the test runner write it: no printed diagnostics
911 /// (they print from the structured form), the generated Lua, and what the check said.
912 pub fn generated(c: &CheckInfo, code: String) -> Self {
913 Self {
914 diagnostics: Vec::new(),
915 errors: c.errors.len(),
916 warnings: c.warnings.len(),
917 lints: c.lints.len(),
918 deps: c.deps.iter().map(|p| normal(p)).collect(),
919 requires: requires_json(c),
920 global_sites: global_sites_json(c),
921 markers: marker_sites_json(c),
922 record_sites: record_sites_json(c),
923 closure_requires: closure_requires_json(c),
924 code: Some(code),
925 check: Some(CheckInfoJson::from_check(c)),
926 }
927 }
928}
929
930/// The directories an entry for `file` probes when there is no project model: the file's
931/// own, the one place a file outside any project resolves its `require`s in
932/// (`project::file_view`).
933///
934/// A store with the model probes the model's answers instead
935/// ([`Cache::with_answers`]), and never reads these. Listed whether or not it exists: a
936/// module appearing beside the file after an entry was written changes what a name
937/// resolves to while every file the entry recorded still hashes the same — the hole
938/// ccache documents in its direct mode.
939pub fn search_dirs(file: &Path) -> Vec<PathBuf> {
940 file.parent().map(Path::to_path_buf).into_iter().collect()
941}
942
943#[derive(Serialize, Deserialize, Debug)]
944struct Entry {
945 stamp: Stamp,
946 /// The file this entry is about, and which question it answers.
947 ///
948 /// Neither is needed to use the entry — the key already decided both. They are here so
949 /// the store can be described: the files on disk are named by hash, and a cache nobody
950 /// can read the contents of is one nobody can reason about (`htl cache status`).
951 subject: String,
952 kind: String,
953 inputs: Vec<Input>,
954 probes: Vec<Probe>,
955 module: Module,
956}
957
958/// One entry holding the whole walk, for `Mode::WholeRun`.
959#[derive(Serialize, Deserialize, Debug)]
960struct RunEntry {
961 stamp: Stamp,
962 /// The files the walk visited, and which question this answers. See `Entry`.
963 subjects: Vec<String>,
964 kind: String,
965 inputs: Vec<Input>,
966 probes: Vec<Probe>,
967 /// One per file in the walk, in walk order. A different count means the walk itself
968 /// changed, which is a miss before any hash is compared.
969 modules: Vec<Module>,
970}
971
972/// One entry holding what `lshape-version` found for the project's declaration and runtime.
973#[derive(Serialize, Deserialize, Debug)]
974struct LshapeEntry {
975 stamp: Stamp,
976 /// The declaration and runtime files this entry read, and which question it answers.
977 /// See `Entry`.
978 subjects: Vec<String>,
979 kind: String,
980 inputs: Vec<Input>,
981 probes: Vec<Probe>,
982 diagnostics: Vec<Recorded>,
983}
984
985/// How much of a run one entry covers.
986///
987/// The two are a real trade rather than one being a refinement of the other, and which
988/// wins depends on where in the dependency graph the edit lands. What `PerModule` saves
989/// depends entirely on that: editing a leaf replays everything else, and editing the
990/// module at the bottom of the graph saves nothing, because everything above it has to be
991/// checked again anyway. Neither mode is slower than a cold check. `WholeRun` keeps one
992/// entry per invocation rather than one per module, which is the reason to reach for it
993/// if the number of files in `.htl/` is the problem. `PerModule` is the default because
994/// editing is the common case.
995#[derive(Copy, Clone, PartialEq, Eq, Debug, Default)]
996pub enum Mode {
997 /// One entry per module. An edit costs the module, its dependents, and what those pull
998 /// in; everything else replays.
999 #[default]
1000 PerModule,
1001 /// One entry for the walk. Any edit anywhere re-checks everything, but a run where
1002 /// nothing moved reads one file instead of one per module.
1003 WholeRun,
1004}
1005
1006impl Mode {
1007 /// The mode a flag or an environment variable names, or `None` for a word this build
1008 /// does not know — which the caller reports as a bad argument rather than silently
1009 /// taking the default for.
1010 pub fn parse(s: &str) -> Option<Self> {
1011 match s {
1012 "per-module" => Some(Mode::PerModule),
1013 "whole-run" => Some(Mode::WholeRun),
1014 _ => None,
1015 }
1016 }
1017
1018 /// The word [`parse`](Self::parse) reads, and the one a run's summary line prints. The
1019 /// two are one spelling on purpose, so what a report says can be pasted back as a flag.
1020 pub fn as_str(self) -> &'static str {
1021 match self {
1022 Mode::PerModule => "per-module",
1023 Mode::WholeRun => "whole-run",
1024 }
1025 }
1026}
1027
1028/// Everything the store needs from outside itself.
1029///
1030/// Built once, by the caller that is already reading flags and the environment, and passed
1031/// in. Nothing in this module reaches for `std::env`: a run's behaviour is decided in one
1032/// place, so reading that place tells you what the run will do.
1033#[derive(Copy, Clone, Debug)]
1034pub struct Options {
1035 /// `--no-cache` turns this off. Separate from how the cache is grained.
1036 pub enabled: bool,
1037 /// How much of a run one entry covers. Held here rather than decided per lookup, so
1038 /// the whole run reads and writes the same grain.
1039 pub mode: Mode,
1040 /// Say why lookups missed, and what the run did with the store.
1041 pub explain: bool,
1042 /// Entries a project's store may hold. `None` scales it to the project size.
1043 pub max_entries: Option<usize>,
1044}
1045
1046impl Default for Options {
1047 fn default() -> Self {
1048 Self {
1049 enabled: true,
1050 mode: Mode::default(),
1051 explain: false,
1052 max_entries: None,
1053 }
1054 }
1055}
1056
1057/// What one run did with the store.
1058///
1059/// Kept by the store rather than by its callers, so a report cannot claim a number the store
1060/// disagrees with.
1061#[derive(Default, Copy, Clone, Debug)]
1062pub struct Stats {
1063 /// Modules replayed from an entry.
1064 pub hits: usize,
1065 /// Modules the store had nothing usable for, so they were checked. Counts a key that
1066 /// was not there and an entry whose inputs had moved alike — both cost the same check,
1067 /// and `explain` is what tells them apart.
1068 pub misses: usize,
1069 /// Entries written. Not the same as `misses`: a run with writing off, or one whose
1070 /// write failed, misses without storing.
1071 pub stored: usize,
1072 /// Entries the sweep dropped, which only `htl check` does.
1073 pub evicted: usize,
1074}
1075
1076impl Stats {
1077 /// One line, for the end of a run. `None` when the run did not touch the store at all,
1078 /// which is worth saying by saying nothing.
1079 pub fn summary(&self, mode: Mode) -> Option<String> {
1080 let touched = self.hits + self.misses + self.stored + self.evicted;
1081 (touched > 0).then(|| {
1082 format!(
1083 "htl cache: {} hit, {} missed, {} written, {} evicted ({})",
1084 self.hits,
1085 self.misses,
1086 self.stored,
1087 self.evicted,
1088 mode.as_str()
1089 )
1090 })
1091 }
1092}
1093
1094/// The identity of one module under one invocation: same key, same question.
1095///
1096/// Carries which question it is, so an entry written under it can record that too — the file
1097/// on disk is named by the hash, and a store whose contents cannot be described is one
1098/// nobody can reason about.
1099#[derive(Debug, Clone)]
1100pub struct Key {
1101 hash: String,
1102 kind: &'static str,
1103}
1104
1105/// Hash of a file's contents, or `None` if it cannot be read — an unreadable input is a
1106/// miss, never a silent hit.
1107fn hash_file(p: &Path) -> Option<String> {
1108 let bytes = std::fs::read(p).ok()?;
1109 Some(blake3::hash(&bytes).to_hex().to_string())
1110}
1111
1112/// A path in the form it is stored and compared in: absolute and symlink-resolved where
1113/// possible.
1114///
1115/// Normalizing at one producer and not another is a real bug rather than a tidiness
1116/// concern: mypy shipped a version where a file checked directly recorded a relative path
1117/// while the same file reached as an import recorded an absolute one, so its hash depended
1118/// on how it was reached. Everything that goes into an entry comes through here.
1119pub fn normal(p: &Path) -> String {
1120 std::fs::canonicalize(p)
1121 .unwrap_or_else(|_| p.to_path_buf())
1122 .to_string_lossy()
1123 .into_owned()
1124}
1125
1126/// Whether `name` resolves to a file in `dir`.
1127///
1128/// The shapes are the ones `H.add_path` puts on the search path (`dir/?.tl`,
1129/// `dir/?/init.tl`) plus the declaration and Lua forms the checker falls back to. A dot in
1130/// a module name is a directory separator, as it is for Lua's own searcher.
1131///
1132/// This is an existence question, not a resolution: it says the name *could* be found here,
1133/// and comparing the answer across every directory on the path is what makes a change of
1134/// resolution visible. Getting it wrong in the direction of "present" costs a re-check;
1135/// there is no direction in which it produces a wrong answer.
1136fn resolves_in(dir: &Path, name: &str) -> bool {
1137 let stem = name.replace('.', "/");
1138 ["tl", "d.tl", "lua"]
1139 .iter()
1140 .any(|ext| dir.join(format!("{stem}.{ext}")).is_file())
1141 || ["init.tl", "init.d.tl", "init.lua"]
1142 .iter()
1143 .any(|f| dir.join(&stem).join(f).is_file())
1144}
1145
1146/// The key for one module in this invocation.
1147///
1148/// `spelling` is the path as the CLI will hand it to the checker, not a resolved one: the
1149/// checker echoes it into every diagnostic it prints, so `htl check .` and
1150/// `htl check src/a.tl` produce different text for the same module and cannot share an
1151/// entry. The working directory completes it, since the same relative spelling means a
1152/// different file from somewhere else.
1153pub fn module_key(spelling: &Path, lint: Option<&str>) -> Key {
1154 key_with(CHECK, spelling, lint)
1155}
1156
1157/// An entry holding what checking a module reported.
1158pub const CHECK: &str = "check";
1159/// An entry holding that, plus the Lua the module generates.
1160pub const GEN: &str = "gen";
1161
1162/// The key for one test file's checked-and-generated form.
1163///
1164/// The same material as [`module_key`] under a different prefix, so the two never collide.
1165/// They are separate entries on purpose: a `htl check` entry holds diagnostics and nothing
1166/// else, and making it carry a few KB of generated Lua would cost every check run a parse it
1167/// has no use for.
1168pub fn gen_key(spelling: &Path, lint: Option<&str>) -> Key {
1169 key_with(GEN, spelling, lint)
1170}
1171
1172/// An entry holding a required module's generated Lua, keyed by the file it is.
1173pub const MODULE: &str = "module";
1174
1175/// The key for a module's checked-and-generated form, by the file it is.
1176///
1177/// A test file is keyed by its spelling ([`gen_key`]) because the runner prints that
1178/// spelling. A module reached through `require` is another matter: the test runner meets
1179/// it by the path the checker resolved, the linker by the one it resolved, `htl build` and
1180/// `include_bundle!` from different working directories — and its generated Lua is the
1181/// same file's Lua whichever way it was reached. So the key is the canonical path and the
1182/// lint selection, and nothing about the invocation, and the entry is stamped without the
1183/// binary (the entry's stamp): that is what lets `htl test`, `htl build` and the macros replay one
1184/// another's entries (#100).
1185pub fn module_gen_key(path: &Path, lint: Option<&str>) -> Key {
1186 let mut h = blake3::Hasher::new();
1187 h.update(MODULE.as_bytes());
1188 h.update(b"\0");
1189 h.update(normal(path).as_bytes());
1190 h.update(b"\0");
1191 h.update(lint.unwrap_or("").as_bytes());
1192 Key {
1193 hash: h.finalize().to_hex().to_string(),
1194 kind: MODULE,
1195 }
1196}
1197
1198/// Whether a module's source could `require` anything: the token appears in it.
1199///
1200/// `gen_lua` comes back without requires for a module the checker already holds — it
1201/// serves the generated code without walking the file again — and a reader that needs the
1202/// requires (the linker's walk, an entry's validation) asks `check` for them. A full check
1203/// for every leaf module is the wrong price for that, and a file that never says
1204/// `require` has nothing to ask about. Word-bounded, so `required_by` does not count.
1205pub fn source_mentions_require(text: &str) -> bool {
1206 let bytes = text.as_bytes();
1207 let ident = |b: u8| b.is_ascii_alphanumeric() || b == b'_';
1208 text.match_indices("require").any(|(i, _)| {
1209 let before = i.checked_sub(1).map(|j| bytes[j]);
1210 let after = bytes.get(i + "require".len()).copied();
1211 !before.is_some_and(ident) && !after.is_some_and(ident)
1212 })
1213}
1214
1215/// The switches every reader of the store honours, from the environment:
1216/// `HTL_NO_CACHE` (neither read nor write), `HTL_CACHE_DEBUG` (say why lookups missed),
1217/// `HTL_CACHE_MAX_ENTRIES` (the sweep's bound; the test suite's, since nothing else can
1218/// reach a few hundred entries by honest means). A command's flags override what this
1219/// read; the proc macros have no flags and take this as it is.
1220impl Options {
1221 /// The options the environment asks for, before any flag is applied.
1222 pub fn from_env() -> Self {
1223 Self {
1224 enabled: std::env::var_os("HTL_NO_CACHE").is_none(),
1225 mode: Mode::PerModule,
1226 explain: std::env::var_os("HTL_CACHE_DEBUG").is_some(),
1227 max_entries: std::env::var_os("HTL_CACHE_MAX_ENTRIES")
1228 .and_then(|v| v.to_str().and_then(|s| s.parse().ok())),
1229 }
1230 }
1231}
1232
1233/// Why nothing may be written under `root`, if nothing may — build scratch, named.
1234///
1235/// A macro expands wherever cargo compiles the crate: in the checkout, but also in the
1236/// copy `cargo publish` verifies under `target/package/<crate>/` — where a new file makes
1237/// cargo abort with "Source directory was modified" — and in a registry checkout under
1238/// `$CARGO_HOME/registry/src`, which nothing should write to. Both are recognisable by
1239/// their path, and neither is anybody's tree to keep anything in.
1240///
1241/// **The rule is the whole `.htl/`, not the store.** Every write htl makes as a side
1242/// effect of reading a project asks here first, and there are two: the run cache
1243/// (`project::store_refusal`, which every reader of the store comes through) and the
1244/// entry links ([`crate::pkg::MluaProject::link_entries`], through
1245/// [`Htl::apply_project`](crate::Htl::apply_project)). The link repair was outside the
1246/// rule until #267, where it wrote `.htl/modules/entries/<dep>` into the tree `cargo
1247/// package` had just built and cargo refused the tarball. A reader of a scratch tree
1248/// reads it exactly as it was packaged.
1249///
1250/// **`target` is recognised by the path and nothing else.** cargo publishes no signal
1251/// saying "this build is the verification step", so htl reads the convention instead: a
1252/// path component named `target`. That has two costs, both accepted for having one rule
1253/// in one function rather than one per writer. A `CARGO_TARGET_DIR` pointing outside the
1254/// tree defeats it — the verify copy is then not under a `target` directory, and htl
1255/// writes in it as it would in a checkout. And a project that keeps its own sources under
1256/// a directory called `target` goes without a store it could have had; a miss costs a
1257/// check, which is the cheaper way to be wrong.
1258pub fn scratch_root(root: &Path) -> Option<&'static str> {
1259 if root.components().any(|c| c.as_os_str() == "target") {
1260 return Some("under a `target` directory");
1261 }
1262 let cargo_home = std::env::var_os("CARGO_HOME")
1263 .map(PathBuf::from)
1264 .or_else(|| std::env::var_os("HOME").map(|h| PathBuf::from(h).join(".cargo")));
1265 if let Some(registry) = cargo_home.map(|h| h.join("registry"))
1266 && root.starts_with(®istry)
1267 {
1268 return Some("a registry checkout");
1269 }
1270 None
1271}
1272
1273fn key_with(kind: &'static str, spelling: &Path, lint: Option<&str>) -> Key {
1274 let mut h = blake3::Hasher::new();
1275 h.update(kind.as_bytes());
1276 h.update(b"\0");
1277 h.update(spelling.to_string_lossy().as_bytes());
1278 h.update(b"\0");
1279 h.update(lint.unwrap_or("").as_bytes());
1280 h.update(b"\0");
1281 h.update(cwd().as_bytes());
1282 Key {
1283 hash: h.finalize().to_hex().to_string(),
1284 kind,
1285 }
1286}
1287
1288/// The key for the walk as a whole, under `Mode::WholeRun`.
1289///
1290/// Every file's spelling goes in, in order: a walk that visits a different set, or the same
1291/// set named differently, is a different run and prints different text.
1292pub fn run_key(files: &[PathBuf], lint: Option<&str>) -> Key {
1293 let mut h = blake3::Hasher::new();
1294 h.update(b"htl-run\0");
1295 for f in files {
1296 h.update(f.to_string_lossy().as_bytes());
1297 h.update(b"\0");
1298 }
1299 h.update(lint.unwrap_or("").as_bytes());
1300 h.update(b"\0");
1301 h.update(cwd().as_bytes());
1302 Key {
1303 hash: h.finalize().to_hex().to_string(),
1304 kind: RUN,
1305 }
1306}
1307
1308/// One entry covering a whole walk (`Mode::WholeRun`).
1309pub const RUN: &str = "run";
1310
1311/// One entry holding what `lshape-version` reported for the project's declaration and
1312/// runtime (#531).
1313pub const LSHAPE: &str = "lshape";
1314
1315/// The key for the project's `lshape-version` verdict.
1316///
1317/// Carries the working directory (findings spell their paths against it) and `LUA_PATH`
1318/// (a declaration with no runtime in the model resolves its runtime on `package.path`,
1319/// #530).
1320pub fn lshape_key() -> Key {
1321 let mut h = blake3::Hasher::new();
1322 h.update(LSHAPE.as_bytes());
1323 h.update(b"\0");
1324 h.update(cwd().as_bytes());
1325 h.update(b"\0");
1326 if let Some(lp) = std::env::var_os("LUA_PATH") {
1327 h.update(lp.as_encoded_bytes());
1328 }
1329 Key {
1330 hash: h.finalize().to_hex().to_string(),
1331 kind: LSHAPE,
1332 }
1333}
1334
1335/// The working directory, resolved once. This is called for every module in the walk, and
1336/// resolving it means a `canonicalize` syscall — doing that per module is a measurable part
1337/// of a run that replays everything and does nothing else.
1338fn cwd() -> &'static str {
1339 static CWD: std::sync::OnceLock<String> = std::sync::OnceLock::new();
1340 CWD.get_or_init(|| {
1341 std::env::current_dir()
1342 .map(|p| normal(&p))
1343 .unwrap_or_default()
1344 })
1345}
1346
1347/// One entry, as `htl cache status` describes it.
1348#[derive(Serialize, Debug)]
1349pub struct EntrySummary {
1350 /// `check`, `gen`, `module`, `run`, `lshape` — or `unreadable` for a file this build
1351 /// cannot parse, which is worth showing rather than hiding, since it is also a permanent
1352 /// miss.
1353 pub kind: String,
1354 /// The files it is about. One, except for a whole-run entry.
1355 pub subjects: Vec<String>,
1356 /// Size of the entry's file on disk, which is what `htl cache status` totals.
1357 pub bytes: u64,
1358 /// Seconds since the entry was last written or replayed. The sweep drops the oldest.
1359 pub age_secs: u64,
1360}
1361
1362/// What a project's store holds.
1363#[derive(Serialize, Debug)]
1364pub struct Contents {
1365 /// The store's directory, said even when it holds nothing — "empty" and "not where you
1366 /// thought" are different answers and a reader cannot tell them apart without it.
1367 pub dir: String,
1368 /// One per entry file. Empty when the directory is missing or unreadable, which are
1369 /// not distinguished: neither is a store this run can use.
1370 pub entries: Vec<EntrySummary>,
1371 /// The whole store's bytes, summed over `entries`.
1372 pub bytes: u64,
1373}
1374
1375/// Read the store and say what is in it.
1376///
1377/// Every entry is parsed, which is why this is a command rather than something a run does on
1378/// the side. Nothing here validates: an entry listed may still be a miss on the next run
1379/// because its inputs moved. This answers "what is stored", not "what would be reused".
1380pub fn describe(root: &Path) -> Contents {
1381 let dir = root.join(DIR);
1382 let mut out = Contents {
1383 dir: dir.to_string_lossy().into_owned(),
1384 entries: Vec::new(),
1385 bytes: 0,
1386 };
1387 let Ok(rd) = std::fs::read_dir(&dir) else {
1388 return out;
1389 };
1390 let now = std::time::SystemTime::now();
1391 for e in rd.flatten() {
1392 let path = e.path();
1393 if path.extension().is_none_or(|x| x != "json") {
1394 continue;
1395 }
1396 let Ok(meta) = e.metadata() else { continue };
1397 out.bytes += meta.len();
1398 let age_secs = meta
1399 .modified()
1400 .ok()
1401 .and_then(|t| now.duration_since(t).ok())
1402 .map(|d| d.as_secs())
1403 .unwrap_or(0);
1404 let v: serde_json::Value = std::fs::read_to_string(&path)
1405 .ok()
1406 .and_then(|raw| serde_json::from_str(&raw).ok())
1407 .unwrap_or(serde_json::Value::Null);
1408 let kind = v
1409 .get("kind")
1410 .and_then(|k| k.as_str())
1411 .unwrap_or("unreadable")
1412 .to_string();
1413 let subjects = match (v.get("subject"), v.get("subjects")) {
1414 (Some(s), _) if s.is_string() => vec![s.as_str().unwrap_or_default().to_string()],
1415 (_, Some(a)) if a.is_array() => a
1416 .as_array()
1417 .unwrap()
1418 .iter()
1419 .filter_map(|x| x.as_str().map(str::to_string))
1420 .collect(),
1421 _ => Vec::new(),
1422 };
1423 out.entries.push(EntrySummary {
1424 kind,
1425 subjects,
1426 bytes: meta.len(),
1427 age_secs,
1428 });
1429 }
1430 out.entries
1431 .sort_by(|a, b| a.kind.cmp(&b.kind).then(a.subjects.cmp(&b.subjects)));
1432 out
1433}
1434
1435/// A store rooted at a project.
1436pub struct Cache {
1437 dir: PathBuf,
1438 opts: Options,
1439 stats: RefCell<Stats>,
1440 /// Content hashes taken during this run, by normalized path.
1441 ///
1442 /// Entries overlap heavily: every module in a project tends to require the same few
1443 /// leaves, so a leaf appears in the inputs of most entries. Without this, checking 48
1444 /// modules against the store reads and hashes the shared ones 48 times — which is most
1445 /// of what a fully replayed run spends. A file is assumed not to change while one
1446 /// `htl check` is running; if it does, the run's answer was undefined anyway.
1447 hashes: RefCell<HashMap<String, Option<String>>>,
1448 /// Name-resolves-here answers taken during this run, by (directory, name). Same
1449 /// reasoning.
1450 dirs: RefCell<HashMap<(String, String), bool>>,
1451 /// The project model's answers, when the caller has a model ([`with_answers`](Self::with_answers)).
1452 answers: Option<Answers>,
1453}
1454
1455impl Cache {
1456 /// `None` when the caller turned caching off, or when this build cannot identify
1457 /// itself well enough to be sure an entry is its own.
1458 pub fn open(root: &Path, opts: Options) -> Option<Self> {
1459 if !opts.enabled {
1460 return None;
1461 }
1462 Stamp::current()?;
1463 Some(Self {
1464 dir: root.join(DIR),
1465 opts,
1466 stats: RefCell::new(Stats::default()),
1467 hashes: RefCell::new(HashMap::new()),
1468 dirs: RefCell::new(HashMap::new()),
1469 answers: None,
1470 })
1471 }
1472
1473 /// Probe with the project model's `answers` instead of the search path's directories.
1474 ///
1475 /// The directories were a stand-in for the question an entry has to ask — does each
1476 /// name the module required still mean the file it meant — and a list of them could
1477 /// only ask it of the directories on the list: a helper added directly under `tests/`
1478 /// was not seen by a test in `tests/sub/`, whose list had `tests/sub` and not `tests`.
1479 /// The model answers the question itself. An entry recorded this way misses in a run
1480 /// without the model, and the reverse, since neither can check the other's probe.
1481 pub fn with_answers(mut self, answers: Answers) -> Self {
1482 self.answers = Some(answers);
1483 self
1484 }
1485
1486 /// The model's answer to each question — a name, asked from the file that required it
1487 /// (`None`: from nowhere in particular, the whole-run entry's own requires) — hashed.
1488 /// The asker is hashed beside the name: the same name asked from two files is two
1489 /// questions, and the model can answer them differently.
1490 fn model_hash(&self, answers: &Answers, questions: &[Question]) -> String {
1491 let mut h = blake3::Hasher::new();
1492 for (asker, n) in questions {
1493 h.update(asker.as_deref().unwrap_or("").as_bytes());
1494 h.update(b"\0");
1495 h.update(n.as_bytes());
1496 h.update(b"\0");
1497 h.update(answers(asker.as_deref().map(Path::new), n).as_bytes());
1498 h.update(b"\x01");
1499 }
1500 h.finalize().to_hex().to_string()
1501 }
1502
1503 /// What this run has done with the store so far.
1504 pub fn stats(&self) -> Stats {
1505 *self.stats.borrow()
1506 }
1507
1508 /// Say why something was not reused. Only when asked; the caller decided that once.
1509 fn miss(&self, reason: &str) {
1510 self.stats.borrow_mut().misses += 1;
1511 if self.opts.explain {
1512 eprintln!("htl cache: miss ({reason})");
1513 }
1514 }
1515
1516 /// The hash of a file, taken once per run.
1517 fn hash_of(&self, path: &str) -> Option<String> {
1518 if let Some(h) = self.hashes.borrow().get(path) {
1519 return h.clone();
1520 }
1521 let h = hash_file(Path::new(path));
1522 self.hashes.borrow_mut().insert(path.to_string(), h.clone());
1523 h
1524 }
1525
1526 /// Whether one name resolves in one directory, answered once per run. The project root,
1527 /// `src/` and `types/` are probed by every module, and modules share the names they
1528 /// require, so this overlaps even more than the file hashes do.
1529 fn resolves(&self, dir: &str, name: &str) -> bool {
1530 let k = (dir.to_string(), name.to_string());
1531 if let Some(v) = self.dirs.borrow().get(&k) {
1532 return *v;
1533 }
1534 let v = resolves_in(Path::new(dir), name);
1535 self.dirs.borrow_mut().insert(k, v);
1536 v
1537 }
1538
1539 /// What `dir` offers for `names`, hashed.
1540 fn probe_hash(&self, dir: &str, names: &[String]) -> String {
1541 let mut h = blake3::Hasher::new();
1542 for n in names {
1543 h.update(n.as_bytes());
1544 h.update(if self.resolves(dir, n) {
1545 b"\x01"
1546 } else {
1547 b"\x00"
1548 });
1549 }
1550 h.finalize().to_hex().to_string()
1551 }
1552
1553 fn entry_path(&self, key: &Key) -> PathBuf {
1554 self.dir.join(format!("{}.json", key.hash))
1555 }
1556
1557 /// The grain this store was opened with. A caller that decides between a per-module
1558 /// walk and a whole-run lookup asks the store rather than re-reading the flags, so the
1559 /// two cannot disagree.
1560 pub fn mode(&self) -> Mode {
1561 self.opts.mode
1562 }
1563
1564 /// A truncated or hand-edited entry is a miss, not a crash.
1565 fn parse<T: serde::de::DeserializeOwned>(&self, raw: &str) -> Option<T> {
1566 match serde_json::from_str(raw) {
1567 Ok(e) => Some(e),
1568 Err(e) => {
1569 self.miss(&format!("unreadable entry: {e}"));
1570 None
1571 }
1572 }
1573 }
1574
1575 /// Every question an entry's probe asks again: each name its modules required, asked
1576 /// from `subject` (the module the entry is about; `None` for a whole-run entry), and
1577 /// each name a member of their require closures required, asked from that member
1578 /// ([`Module::closure_requires`]). Sorted and deduplicated.
1579 ///
1580 /// Both sides of a probe comparison have to derive this the same way, which is why it is
1581 /// one function rather than two loops.
1582 fn questions(modules: &[Module], subject: Option<&str>) -> Vec<Question> {
1583 let mut qs: Vec<Question> = modules
1584 .iter()
1585 .flat_map(|m| {
1586 let own = m
1587 .requires
1588 .iter()
1589 .map(|r| (subject.map(str::to_string), r.module.clone()));
1590 let below = m
1591 .closure_requires
1592 .iter()
1593 .map(|e| (Some(e.from.clone()), e.module.clone()));
1594 own.chain(below)
1595 })
1596 .collect();
1597 qs.sort();
1598 qs.dedup();
1599 qs
1600 }
1601
1602 /// The names the questions ask, whoever asks them, sorted and deduplicated: what a
1603 /// directory probe hashes, since a directory offers a name or does not whoever asks.
1604 fn names_of(questions: &[Question]) -> Vec<String> {
1605 let mut names: Vec<String> = questions.iter().map(|(_, n)| n.clone()).collect();
1606 names.sort();
1607 names.dedup();
1608 names
1609 }
1610
1611 /// Whether an entry still describes the world: written by this build, every file it read
1612 /// unchanged, and every directory it could have resolved in still answering the same way
1613 /// for the names it asked about.
1614 fn still_valid(
1615 &self,
1616 kind: &str,
1617 stamp: &Stamp,
1618 inputs: &[Input],
1619 probes: &[Probe],
1620 questions: &[Question],
1621 ) -> bool {
1622 let Some(current) = Stamp::for_kind(kind) else {
1623 return false;
1624 };
1625 if *stamp != current {
1626 self.miss("written by a different build");
1627 return false;
1628 }
1629 for i in inputs {
1630 match self.hash_of(&i.path) {
1631 Some(h) if h == i.hash => {}
1632 Some(_) => {
1633 self.miss(&format!("changed: {}", i.path));
1634 return false;
1635 }
1636 None => {
1637 self.miss(&format!("gone: {}", i.path));
1638 return false;
1639 }
1640 }
1641 }
1642 for p in probes {
1643 if p.dir == MODEL_PROBE {
1644 let Some(answers) = &self.answers else {
1645 self.miss("recorded against a project model this run does not have");
1646 return false;
1647 };
1648 if self.model_hash(answers, questions) != p.names {
1649 self.miss(
1650 "what a name this module or its require closure requires resolves to changed",
1651 );
1652 return false;
1653 }
1654 continue;
1655 }
1656 if self.answers.is_some() {
1657 self.miss("recorded without the project model this run has");
1658 return false;
1659 }
1660 if self.probe_hash(&p.dir, &Self::names_of(questions)) != p.names {
1661 self.miss(&format!(
1662 "what {} offers for this module's requires changed",
1663 p.dir
1664 ));
1665 return false;
1666 }
1667 }
1668 self.stats.borrow_mut().hits += 1;
1669 true
1670 }
1671
1672 /// What each file in the walk reported last time; `None` where it has to be checked.
1673 ///
1674 /// Under `WholeRun` this is all-or-nothing by construction: one entry covers the walk,
1675 /// so a single changed input means every module is checked.
1676 pub fn lookup_all(&self, keys: &[Key], run: &Key, files: usize) -> Vec<Option<Module>> {
1677 match self.opts.mode {
1678 Mode::PerModule => keys.iter().map(|k| self.lookup(k)).collect(),
1679 Mode::WholeRun => match self.lookup_run(run, files) {
1680 Some(ms) => ms.into_iter().map(Some).collect(),
1681 None => vec![None; files],
1682 },
1683 }
1684 }
1685
1686 /// What the module reported last time, if every input and probe still matches.
1687 pub fn lookup(&self, key: &Key) -> Option<Module> {
1688 let raw = std::fs::read_to_string(self.entry_path(key)).ok()?;
1689 let entry: Entry = self.parse(&raw)?;
1690 let questions = Self::questions(std::slice::from_ref(&entry.module), Some(&entry.subject));
1691 if !self.still_valid(
1692 &entry.kind,
1693 &entry.stamp,
1694 &entry.inputs,
1695 &entry.probes,
1696 &questions,
1697 ) {
1698 return None;
1699 }
1700 self.touch(key);
1701 Some(entry.module)
1702 }
1703
1704 /// Mark an entry as used, so that the sweep drops what is stale rather than what is
1705 /// merely old.
1706 ///
1707 /// Without this the shape you run most often is the one whose entries were written
1708 /// first, so it ages out while a lint flag you tried once survives — measured on a
1709 /// 58-module project, running four other shapes left the plain one replaying 24 of its
1710 /// 58 modules. Failing to touch costs a later re-check and nothing else, so every error
1711 /// here is ignored.
1712 fn touch(&self, key: &Key) {
1713 if let Ok(f) = std::fs::File::options()
1714 .write(true)
1715 .open(self.entry_path(key))
1716 {
1717 let _ = f.set_modified(std::time::SystemTime::now());
1718 }
1719 }
1720
1721 fn lookup_run(&self, key: &Key, files: usize) -> Option<Vec<Module>> {
1722 let raw = std::fs::read_to_string(self.entry_path(key)).ok()?;
1723 let entry: RunEntry = self.parse(&raw)?;
1724 if entry.modules.len() != files {
1725 self.miss("the walk visits a different number of files");
1726 return None;
1727 }
1728 let questions = Self::questions(&entry.modules, None);
1729 if !self.still_valid(RUN, &entry.stamp, &entry.inputs, &entry.probes, &questions) {
1730 return None;
1731 }
1732 self.touch(key);
1733 Some(entry.modules)
1734 }
1735
1736 /// Hash every path an entry has to compare next time, once each: a dependency reached
1737 /// from two modules is compared once. `None` if anything is unreadable, which drops the
1738 /// entry rather than storing one that would hit wrongly.
1739 fn inputs_for(&self, paths: impl Iterator<Item = String>) -> Option<Vec<Input>> {
1740 let mut paths: Vec<String> = paths.collect();
1741 paths.sort();
1742 paths.dedup();
1743 let mut inputs = Vec::with_capacity(paths.len());
1744 for p in paths {
1745 inputs.push(Input {
1746 path: p.clone(),
1747 hash: self.hash_of(&p)?,
1748 });
1749 }
1750 Some(inputs)
1751 }
1752
1753 /// The probes an entry records for `questions`: the model's answers, when this store has
1754 /// the model; otherwise one per directory — `dirs`, the caller's for the module, and the
1755 /// directory of every member of the closure that asked something, which is where a file
1756 /// outside any project resolves its own `require`s ([`search_dirs`]).
1757 fn probes_for(&self, dirs: &[PathBuf], questions: &[Question]) -> Vec<Probe> {
1758 if let Some(answers) = &self.answers {
1759 return vec![Probe {
1760 dir: MODEL_PROBE.to_string(),
1761 names: self.model_hash(answers, questions),
1762 }];
1763 }
1764 let askers = questions
1765 .iter()
1766 .filter_map(|(asker, _)| asker.as_deref())
1767 .flat_map(|a| search_dirs(Path::new(a)));
1768 let mut dirs: Vec<String> = dirs
1769 .iter()
1770 .cloned()
1771 .chain(askers)
1772 .map(|p| normal(&p))
1773 .collect();
1774 dirs.sort();
1775 dirs.dedup();
1776 let names = Self::names_of(questions);
1777 dirs.into_iter()
1778 .map(|d| {
1779 let h = self.probe_hash(&d, &names);
1780 Probe { dir: d, names: h }
1781 })
1782 .collect()
1783 }
1784
1785 /// Record what one module reported. Best-effort: a store that cannot be written leaves
1786 /// the next run to do the work again, which is slow rather than wrong.
1787 pub fn store_module(
1788 &self,
1789 key: &Key,
1790 file: &Path,
1791 extra_inputs: &[PathBuf],
1792 dirs: &[PathBuf],
1793 module: &Module,
1794 ) {
1795 let Some(stamp) = Stamp::for_kind(key.kind) else {
1796 return;
1797 };
1798 let paths = std::iter::once(normal(file))
1799 .chain(module.deps.iter().cloned())
1800 .chain(extra_inputs.iter().map(|p| normal(p)));
1801 let Some(inputs) = self.inputs_for(paths) else {
1802 return;
1803 };
1804 let subject = normal(file);
1805 let questions = Self::questions(std::slice::from_ref(module), Some(&subject));
1806 let entry = Entry {
1807 stamp,
1808 subject: subject.clone(),
1809 kind: key.kind.to_string(),
1810 inputs,
1811 probes: self.probes_for(dirs, &questions),
1812 module: module.clone(),
1813 };
1814 match self.write(key, &entry) {
1815 Ok(()) => self.stats.borrow_mut().stored += 1,
1816 Err(e) if self.opts.explain => eprintln!("htl cache: not stored ({e})"),
1817 Err(_) => {}
1818 }
1819 }
1820
1821 /// Record the whole walk as one entry. Its inputs are the union of every module's, so
1822 /// any edit anywhere misses — which is the behaviour this mode is chosen for.
1823 pub fn store_run(
1824 &self,
1825 key: &Key,
1826 files: &[PathBuf],
1827 extra_inputs: &[PathBuf],
1828 dirs: &[PathBuf],
1829 modules: &[Module],
1830 ) {
1831 let Some(stamp) = Stamp::current() else {
1832 return;
1833 };
1834 let paths = files
1835 .iter()
1836 .map(|f| normal(f))
1837 .chain(modules.iter().flat_map(|m| m.deps.iter().cloned()))
1838 .chain(extra_inputs.iter().map(|p| normal(p)));
1839 let Some(inputs) = self.inputs_for(paths) else {
1840 return;
1841 };
1842 let questions = Self::questions(modules, None);
1843 let entry = RunEntry {
1844 stamp,
1845 subjects: files.iter().map(|f| normal(f)).collect(),
1846 kind: key.kind.to_string(),
1847 inputs,
1848 probes: self.probes_for(dirs, &questions),
1849 modules: modules.to_vec(),
1850 };
1851 match self.write(key, &entry) {
1852 Ok(()) => self.stats.borrow_mut().stored += 1,
1853 Err(e) if self.opts.explain => eprintln!("htl cache: not stored ({e})"),
1854 Err(_) => {}
1855 }
1856 }
1857
1858 /// What `lshape-version` reported last time, if the declaration, the runtime, the
1859 /// project model's answer for `lshape` and any `package.path` directories it consulted
1860 /// still match.
1861 ///
1862 /// Does not touch [`Stats`]: those count the walk's modules.
1863 pub fn lookup_lshape(&self, key: &Key) -> Option<Vec<crate::Diagnostic>> {
1864 let raw = std::fs::read_to_string(self.entry_path(key)).ok()?;
1865 let entry: LshapeEntry = serde_json::from_str(&raw).ok()?;
1866 if entry.kind != LSHAPE || entry.stamp != Stamp::for_kind(LSHAPE)? {
1867 return None;
1868 }
1869 for i in &entry.inputs {
1870 if self.hash_of(&i.path).as_deref() != Some(i.hash.as_str()) {
1871 return None;
1872 }
1873 }
1874 let questions = vec![(None, "lshape".to_string())];
1875 let names = Self::names_of(&questions);
1876 for p in &entry.probes {
1877 if p.dir == MODEL_PROBE {
1878 let answers = self.answers.as_ref()?;
1879 if self.model_hash(answers, &questions) != p.names {
1880 return None;
1881 }
1882 } else if self.probe_hash(&p.dir, &names) != p.names {
1883 return None;
1884 }
1885 }
1886 let mut out = Vec::with_capacity(entry.diagnostics.len());
1887 for r in &entry.diagnostics {
1888 let severity = crate::Severity::parse(&r.severity)?;
1889 let fix = r.fix.as_ref().map(FixJson::to_fix);
1890 out.push(r.item.to_diagnostic(severity, fix));
1891 }
1892 self.touch(key);
1893 Some(out)
1894 }
1895
1896 /// Record what `lshape-version` reported for `files` (the declaration and, when found,
1897 /// the runtime) and `dirs` (the search path directories `package.path` consulted when
1898 /// the model held a declaration alone).
1899 pub fn store_lshape(
1900 &self,
1901 key: &Key,
1902 files: &[PathBuf],
1903 dirs: &[PathBuf],
1904 diagnostics: &[crate::Diagnostic],
1905 ) {
1906 if files.is_empty() {
1907 return;
1908 }
1909 let Some(stamp) = Stamp::for_kind(key.kind) else {
1910 return;
1911 };
1912 let Some(inputs) = self.inputs_for(files.iter().map(|f| normal(f))) else {
1913 return;
1914 };
1915 let questions = vec![(None, "lshape".to_string())];
1916 let names = Self::names_of(&questions);
1917 let mut probes = Vec::new();
1918 if let Some(answers) = &self.answers {
1919 probes.push(Probe {
1920 dir: MODEL_PROBE.to_string(),
1921 names: self.model_hash(answers, &questions),
1922 });
1923 }
1924 let mut norm_dirs: Vec<String> = dirs.iter().map(|d| normal(d)).collect();
1925 norm_dirs.sort();
1926 norm_dirs.dedup();
1927 for d in norm_dirs {
1928 let h = self.probe_hash(&d, &names);
1929 probes.push(Probe { dir: d, names: h });
1930 }
1931 let recorded = diagnostics
1932 .iter()
1933 .map(|d| Recorded {
1934 severity: d.severity.as_str().to_string(),
1935 item: ItemJson::from_diagnostic(d),
1936 fix: d.fix.as_ref().map(FixJson::from_fix),
1937 dependency: None,
1938 })
1939 .collect();
1940 let entry = LshapeEntry {
1941 stamp,
1942 subjects: files.iter().map(|f| normal(f)).collect(),
1943 kind: key.kind.to_string(),
1944 inputs,
1945 probes,
1946 diagnostics: recorded,
1947 };
1948 let _ = self.write(key, &entry);
1949 }
1950
1951 /// Drop entries this run did not use, once the store has outgrown what the project
1952 /// warrants.
1953 ///
1954 /// Nothing else removes an entry. The key covers the paths as written, the lint
1955 /// selection and the working directory, so every distinct way of invoking the check
1956 /// leaves a full set of module entries behind — a lint flag tried once doubles the store
1957 /// permanently, and a deleted module's entry is never read again. `htl check` sees the
1958 /// whole graph in one process, which is what lets this be exact about the orphans rather
1959 /// than sampling the way ccache has to.
1960 ///
1961 /// Two passes. Entries whose recorded inputs have all gone describe modules that no
1962 /// longer exist, and go first. If the store is still over, the oldest go until it fits.
1963 ///
1964 /// **This uses mtimes, and #3's rule against them still holds.** That rule is about
1965 /// invalidation, where trusting a timestamp means replaying a stale result and reporting
1966 /// something untrue. Dropping an entry that was still good costs the check it would have
1967 /// skipped and nothing else. The two questions deserve different tools.
1968 ///
1969 /// "Oldest" is least recently *used*: a hit touches its entry, which is what makes the
1970 /// two differ.
1971 /// Without that it would mean least recently written, and the shape run most often —
1972 /// written first — would age out while a shape tried once survived.
1973 pub fn sweep(&self, keep: &[Key], files: usize) {
1974 let bound = self
1975 .opts
1976 .max_entries
1977 .unwrap_or_else(|| default_bound(files));
1978 let Ok(rd) = std::fs::read_dir(&self.dir) else {
1979 return;
1980 };
1981 let keep: std::collections::HashSet<&str> = keep.iter().map(|k| k.hash.as_str()).collect();
1982
1983 let mut all = 0usize;
1984 let mut candidates: Vec<(PathBuf, std::time::SystemTime)> = Vec::new();
1985 for e in rd.flatten() {
1986 let path = e.path();
1987 if path.extension().is_none_or(|x| x != "json") {
1988 continue;
1989 }
1990 all += 1;
1991 let stem = path
1992 .file_stem()
1993 .map(|s| s.to_string_lossy().into_owned())
1994 .unwrap_or_default();
1995 if keep.contains(stem.as_str()) {
1996 continue;
1997 }
1998 let mtime = e
1999 .metadata()
2000 .and_then(|m| m.modified())
2001 .unwrap_or(std::time::UNIX_EPOCH);
2002 candidates.push((path, mtime));
2003 }
2004 if all <= bound {
2005 return;
2006 }
2007
2008 let mut removed = 0usize;
2009 // Orphans first: an entry none of whose inputs still exist cannot be read again.
2010 candidates.retain(|(p, _)| {
2011 if all - removed <= bound || !self.is_orphan(p) {
2012 return true;
2013 }
2014 if std::fs::remove_file(p).is_ok() {
2015 removed += 1;
2016 }
2017 false
2018 });
2019 // Then by age, oldest first, until the store fits.
2020 if all - removed > bound {
2021 candidates.sort_by_key(|(_, t)| *t);
2022 for (p, _) in &candidates {
2023 if all - removed <= bound {
2024 break;
2025 }
2026 if std::fs::remove_file(p).is_ok() {
2027 removed += 1;
2028 }
2029 }
2030 }
2031 self.stats.borrow_mut().evicted += removed;
2032 if removed > 0 && self.opts.explain {
2033 eprintln!("htl cache: dropped {removed} of {all} entries (bound {bound})");
2034 }
2035 }
2036
2037 /// Whether every file an entry recorded as an input has gone. Unreadable entries count as
2038 /// orphans: nothing can use them either.
2039 fn is_orphan(&self, path: &Path) -> bool {
2040 let Ok(raw) = std::fs::read_to_string(path) else {
2041 return true;
2042 };
2043 let Ok(v) = serde_json::from_str::<serde_json::Value>(&raw) else {
2044 return true;
2045 };
2046 let Some(inputs) = v.get("inputs").and_then(|i| i.as_array()) else {
2047 return true;
2048 };
2049 !inputs
2050 .iter()
2051 .filter_map(|i| i.get("path").and_then(|p| p.as_str()))
2052 .any(|p| Path::new(p).exists())
2053 }
2054
2055 /// Whole or not at all: a temporary file in the same directory, then a rename. On every
2056 /// filesystem htl runs on that rename is atomic, so a concurrent reader sees either the
2057 /// previous entry or this one, never half of this one. ccache relies on exactly this
2058 /// and takes no locks at all.
2059 fn write<T: Serialize>(&self, key: &Key, entry: &T) -> Result<()> {
2060 std::fs::create_dir_all(&self.dir)?;
2061 let final_path = self.entry_path(key);
2062 let tmp = self
2063 .dir
2064 .join(format!(".{}.{}.tmp", key.hash, std::process::id()));
2065 std::fs::write(&tmp, serde_json::to_vec(entry)?)?;
2066 if let Err(e) = std::fs::rename(&tmp, &final_path) {
2067 let _ = std::fs::remove_file(&tmp);
2068 return Err(e.into());
2069 }
2070 Ok(())
2071 }
2072}
2073
2074#[cfg(test)]
2075mod tests {
2076 use super::*;
2077 use crate::core_common;
2078
2079 fn tempdir(name: &str) -> core_common::TempDir {
2080 let dir = core_common::tempdir("htl-core-cache", name);
2081 std::fs::create_dir_all(dir.join("src")).unwrap();
2082 dir
2083 }
2084
2085 /// A `module` entry is stamped without the binary, so one written by `htl` is served
2086 /// inside a proc macro (whose binary is `rustc`) and the other way round. A test-file
2087 /// entry keeps the binary in its stamp, as it always did.
2088 #[test]
2089 fn a_module_entry_is_served_whatever_binary_wrote_it() {
2090 let root = tempdir("portable");
2091 let file = root.join("src/m.tl");
2092 std::fs::write(&file, "return {}\n").unwrap();
2093 let cache = Cache::open(&root, Options::default()).unwrap();
2094 // What another binary writes for a module: the portable stamp. This binary's own
2095 // stamp is a different value, which is the point.
2096 assert_ne!(Stamp::current().unwrap(), Stamp::portable());
2097 let module = Module {
2098 diagnostics: Vec::new(),
2099 errors: 0,
2100 warnings: 0,
2101 lints: 0,
2102 deps: Vec::new(),
2103 requires: Vec::new(),
2104 global_sites: Vec::new(),
2105 markers: Vec::new(),
2106 record_sites: Vec::new(),
2107 closure_requires: Vec::new(),
2108 code: Some("return {}".into()),
2109 check: Some(CheckInfoJson::default()),
2110 };
2111 let entry = |kind: &str| Entry {
2112 stamp: Stamp::portable(),
2113 subject: normal(&file),
2114 kind: kind.to_string(),
2115 inputs: cache.inputs_for(std::iter::once(normal(&file))).unwrap(),
2116 probes: Vec::new(),
2117 module: module.clone(),
2118 };
2119 let mk = module_gen_key(&file, None);
2120 cache.write(&mk, &entry(MODULE)).unwrap();
2121 assert!(
2122 cache.lookup(&mk).is_some(),
2123 "a module entry: the binary is not part of its stamp"
2124 );
2125 let gk = gen_key(&file, None);
2126 cache.write(&gk, &entry(GEN)).unwrap();
2127 assert!(
2128 cache.lookup(&gk).is_none(),
2129 "a test-file entry from another binary: written by a different build"
2130 );
2131 // And what this binary stores under a module key is what any other reads.
2132 cache.store_module(&mk, &file, &[], &[], &module);
2133 let raw = std::fs::read_to_string(cache.entry_path(&mk)).unwrap();
2134 let stored: Entry = serde_json::from_str(&raw).unwrap();
2135 assert_eq!(stored.stamp, Stamp::portable());
2136 }
2137
2138 #[test]
2139 fn source_mentions_require_is_word_bounded() {
2140 assert!(source_mentions_require("local a = require(\"a\")\n"));
2141 assert!(source_mentions_require("require 'a'"));
2142 assert!(!source_mentions_require("local required_by = 1\n"));
2143 assert!(!source_mentions_require("local x = prerequire\n"));
2144 assert!(!source_mentions_require("local record c\nend\nreturn c\n"));
2145 assert!(!source_mentions_require(""));
2146 }
2147
2148 #[test]
2149 fn scratch_roots_are_recognised() {
2150 assert_eq!(
2151 scratch_root(Path::new("/w/target/package/host-0.1.0")),
2152 Some("under a `target` directory")
2153 );
2154 assert_eq!(scratch_root(Path::new("/w/host")), None);
2155 let cargo_home = std::env::var_os("CARGO_HOME")
2156 .map(PathBuf::from)
2157 .or_else(|| std::env::var_os("HOME").map(|h| PathBuf::from(h).join(".cargo")))
2158 .expect("CARGO_HOME or HOME");
2159 assert_eq!(
2160 scratch_root(&cargo_home.join("registry/src/index/htl-0.3.0")),
2161 Some("a registry checkout")
2162 );
2163 }
2164}