znippy_plugin_git/resolve.rs
1//! Pack entries → **oids**, which is the one thing [`crate::pack_walk`] cannot
2//! tell you and the `objects` table is keyed by.
3//!
4//! This is the **indexer's** work, not the ack path's (§13.9: "the index is built
5//! after the ack, over a channel"). It inflates every entry, applies every delta
6//! chain and hashes the result, which is what `git index-pack` does and what
7//! makes it the expensive half of a push. Nothing here is called before a client
8//! is told its push landed.
9//!
10//! # What it keeps and what it throws away
11//!
12//! Peak memory is deliberately not "the whole history inflated". **An object is
13//! held only while an unresolved entry still has to read it**, and for nothing
14//! else:
15//!
16//! | object | held | until |
17//! |---|---|---|
18//! | a delta base | yes | its **last** dependent has consumed it |
19//! | everything else — blob, commit, tree, tag | **no** | it leaves through the [`PayloadSink`] and is dropped |
20//!
21//! The base set and, more to the point, **how many entries read each base** are
22//! known before a byte is inflated: the walk already collected every delta base,
23//! so the resolver starts with an exact reference count per base and drops a
24//! payload the moment that count reaches zero. The last dependent gets the base
25//! **moved** into it rather than cloned, so a linear delta chain — one base, one
26//! dependent, the ordinary shape of real history — copies no payload at all.
27//!
28//! ## What this table said before 2026-08-11, and what it cost
29//!
30//! It said commits, trees and tags were kept "because the `graph` and `reach`
31//! tables are built from them". They are not, and had not been since §14's
32//! exploded table became the fold's input — `absorb_one` reads
33//! `Resolved::index_entry()` and nothing else. Two copies of the object set were
34//! being held for a reader that no longer existed:
35//!
36//! * `content` kept every base **and** every typed object for the whole call.
37//! Nothing ever removed a row, so "held only while something still needs them"
38//! described an intention, not the code.
39//! * `Resolved::payload` kept a second copy of the same bytes, cloned in.
40//!
41//! MEASURED on oden 2026-08-11, `tests/resolve_peak_memory.rs`, peak **live**
42//! heap over `resolve_walked` against the pack's own inflated payload:
43//!
44//! | pack | before | after |
45//! |---|---:|---:|
46//! | forge-year-8a-240b-12c, 28 893 objects, 111.0 MB | 2.03x | **0.03x** |
47//! | stage-red-nobitmap, 12 328 objects, 343.5 MB | 1.34x (461.2 MB) | **0.02x** (5.9 MB) |
48//! | facett, 7 232 objects, 72.8 MB | 1.20x | **0.42x** |
49//! | znippy's own, 5 388 objects, 84.4 MB | 0.75x | **0.14x** |
50//!
51//! The worst pack in the corpus went 2.03x → 0.42x. It is `facett`, whose
52//! delta chains fan out widely enough that a real base working set is genuinely
53//! live at once — which is the bound behaving as intended rather than a residue.
54//!
55//! End to end, one `gunnar-server`, one `push --mirror`, anonymous RSS held
56//! afterwards (not peak — this memory was never given back):
57//!
58//! | fixture | before | after |
59//! |---|---:|---:|
60//! | 36 172 objects, 290 MB inflated, 5.9 MB pack | 575.7 MB | **185.1 MB** |
61//! | 352 396 objects, 2.89 GB inflated, 61.8 MB pack | 4480.1 MB | **439.9 MB** |
62//!
63//! The second row is the one that mattered: 10x the repository cost 10x the
64//! memory before and 2.4x after, so the resolver's footprint stopped tracking
65//! the history. What is left is bounded and named — redb's page cache
66//! (`ZNIPPY_GIT_REDB_CACHE_BYTES`, 64 MiB by default, measured at 129.3 MB held
67//! with an 8 MiB ceiling and 303.9 MB with 256 MiB), the exploded table's
68//! [`FLUSH_BYTES`](crate::exploded::FLUSH_BYTES) write buffer, and
69//! `Derived`'s in-memory graph, which is genuinely O(repository).
70//!
71//! # ⚠ The clone that got slower, and why it is not this code
72//!
73//! **Read this before "fixing" the reference counting.** `vs_forge_clone` on the
74//! 352 396-object fixture reported this change as a **+78 % CPU regression on
75//! every znippy arm**, and the natural reading — a base dropped at its "last
76//! use" has to be re-resolved when something later wants it again — is wrong. It
77//! is not even mechanically possible: an undercount cannot cause a re-resolve
78//! here, because there is no re-resolve to fall into. `resolve_walked` fails the
79//! whole pack by name.
80//!
81//! What it actually is, MEASURED on oden 2026-08-11, one gunnar-server, seed the
82//! fixture then clone it, steady state (clones 2-3), quiet box, only the
83//! allocator and this file varying:
84//!
85//! | resolver | allocator | clone CPU | anon after seed |
86//! |---|---|---:|---:|
87//! | before | glibc | 1.47 s | 4481 MB |
88//! | **after** | glibc | **2.62 s** | **438 MB** |
89//! | before | mimalloc | 1.18 s | 1767 MB |
90//! | **after** | **mimalloc** | **1.17 s** | **701 MB** |
91//!
92//! With a competent allocator the two resolvers are **identical** (1.18 against
93//! 1.17). The regression is entirely glibc `malloc`: the old resolver's retained
94//! 4.5 GB was accidentally acting as a **pre-warmed pool**, so the clone's own
95//! ~2 GB of allocations came off a free list instead of out of the kernel. Take
96//! the leak away and glibc charges for the memory it should have been charging
97//! for all along.
98//!
99//! Three independent measurements say the same thing, and none of them involve
100//! this file being slower:
101//!
102//! 1. **Bisect.** A build with *only* `Resolved::payload` removed — the
103//! reference counting NOT applied, `content` still never pruned — reproduces
104//! the whole regression (2.56–2.62 s). Adding the reference counting on top
105//! then costs **nothing** (2.55–2.65 s) and takes anon-after-seed from
106//! 2350 MB to 438 MB. The counting is free; the *not leaking* is what glibc
107//! punishes.
108//! 2. **Restart.** Restart the server after seeding, so the derived tables are
109//! folded onto a fresh heap: before 2.20–2.22 s, after 2.20–2.24 s —
110//! identical to within 1 %. The old resolver's 1.47 s is the outlier, not the
111//! new one's 2.62 s.
112//! 3. **The resolver itself.** Over the same 6 real packs both revisions accept,
113//! 1.058 s before against 0.975 s after: this code is ~8 % *faster*.
114//!
115//! So there is no cheaper trade to find inside this module, and an LRU over
116//! bases would buy nothing. The lever, if the CPU is wanted back, is the
117//! allocator — and `LD_PRELOAD=libmimalloc.so.3` is enough to get it with no
118//! code change at all.
119//!
120//! # Where the payloads it throws away now go
121//!
122//! §14's exploded table is **eager** (decided 2026-08-08), so every object this
123//! module resolves has to reach [`crate::exploded::ExplodedTable`] — including
124//! the blobs the table above says are dropped. They still are: the payload
125//! leaves through a [`PayloadSink`] as it is produced, rather than being
126//! accumulated into the returned `Vec`. That is the difference between a peak
127//! footprint of "the bases plus the typed objects" and one of "the whole pack
128//! inflated at once", and eager resolution does not get to change it.
129//!
130//! A **thin** pack's `REF_DELTA` base is an object the client knows the server
131//! already has, and it is usually a blob. Resolving it needs that blob's
132//! *content*, which is exactly what the exploded table now holds — so the
133//! [`BaseSource`] can answer from it in one point lookup instead of re-resolving
134//! a whole pack. A thin pack whose external base is **still** not re-derivable
135//! is refused by name ([`ResolveError::MissingBase`]) rather than resolved by
136//! guesswork, unchanged.
137//!
138//! # The bounded base set (2026-08-30)
139//!
140//! The table above says a base is held until its last dependent. On
141//! torvalds/linux that set is **unbounded in principle** — 11.2 M deltas over
142//! 13.9 M objects, and nothing in the pack format promises a base's last
143//! dependent is near it. On the 16 GB appliance the indexer grew past the box
144//! and the kernel panicked (see [`crate::memguard`]). So the set now has a
145//! **byte ceiling** ([`ResolveLimits::base_budget`], default 25 % of
146//! `MemTotal`, `ZNIPPY_GIT_BASE_BUDGET_BYTES`):
147//!
148//! | event | what happens |
149//! |---|---|
150//! | a base would push the held bytes over the ceiling | the held base whose **next dependent is farthest** in the pack is dropped (Belady — the walk already lists every dependent's position, so "next use" is known, not guessed) |
151//! | the new base itself is the farthest | it is not held at all |
152//! | a dependent asks for a dropped base | it is **re-inflated from the mmapped pack** — its own chain re-applied — and counted in [`ResolveReport::reinflations`] |
153//! | the watchdog says the box is short | **every** held base is dropped, the sink and the row batch are flushed, and the worker waits |
154//!
155//! Same output, hard ceiling. The ceiling is `max(budget, largest single
156//! base)`: one object bigger than the budget is held while it is needed,
157//! because the alternative is re-inflating it once per dependent.
158//!
159//! # What is preallocated
160//!
161//! Two scratch buffers live across the whole pass: the delta instruction
162//! stream and the payload of an object nothing deltas against. An object that
163//! is a base gets its own `Vec` because it outlives the iteration; every other
164//! object — the overwhelming majority — is inflated into the reused buffer,
165//! hashed in place ([`GitHashKind::oid_of_object`], no canonical copy) and
166//! handed to the sink from there. Per object the hot path allocates the
167//! 20-byte oid of its row and nothing else.
168
169use std::collections::HashMap;
170use std::sync::Arc;
171use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
172
173use anyhow::{Result, anyhow, bail};
174
175use crate::exploded::{NoSink, PayloadSink};
176use crate::forest::{DeltaForest, EntrySet};
177use crate::index_layout::{IndexEntry, ObjType};
178use crate::memguard::MemWatchdog;
179use crate::object::{GitHashKind, GitObjectKind};
180use crate::pack_walk::{DeltaBase, PackEntry, PackWalk, walk};
181use crate::progress::{IndexProgress, Phase};
182
183/// Where an entry that deltas against something outside the pack gets its base.
184///
185/// Implemented by the store, which may or may not be able to answer — see the
186/// module docs on §14.
187pub trait BaseSource: Sync {
188 /// The object's type and inflated payload.
189 ///
190 /// `Ok(None)` means **this source does not hold that object** — the only
191 /// answer [`ResolveError::missing_base`] may be built from. `Err` means the
192 /// source *does* hold it and could not read it, and that is a different
193 /// fact with a different owner.
194 ///
195 /// # 🔴 It returned a bare `Option`, and the difference was thrown away
196 ///
197 /// `impl BaseSource for Absorber` ended in `.ok().flatten()`: a store that
198 /// failed to reconstruct a base it holds answered exactly like a store that
199 /// never had it, and the resolver then refused the *pushed pack* with *"this
200 /// repository genuinely does not have that object"*. MEASURED on oden
201 /// 2026-08-30 (lane AI): 21 of 140 ordinary pushes refused with that
202 /// sentence about an object sitting in an absorbed pack one lookup away —
203 /// see [`MAX_DELTA_LINKS`] for what was actually wrong and
204 /// `crate::thin_storm` for the red. A client cannot act on a diagnosis that
205 /// names the wrong party; neither can an operator.
206 fn content(&self, oid: &[u8]) -> Result<Option<(GitObjectKind, Vec<u8>)>>;
207}
208
209/// A source that has nothing. The right one for a self-contained pack, and it
210/// makes "this pack was thin" an error rather than a silent success.
211pub struct NoBases;
212
213impl BaseSource for NoBases {
214 fn content(&self, _oid: &[u8]) -> Result<Option<(GitObjectKind, Vec<u8>)>> {
215 Ok(None)
216 }
217}
218
219/// One resolved object: **exactly what the `objects` table stores**, and no
220/// payload.
221///
222/// It carried an `Option<Vec<u8>>` payload until 2026-08-11, for a reader that
223/// stopped existing when §14's exploded table became the fold's input. Every
224/// production caller uses [`index_entry`](Self::index_entry) and nothing else —
225/// `absorb_one` builds index rows, and the one other caller discards the vector
226/// entirely and reads its sink. Content has one home now
227/// ([`PayloadSink`], LAW 5) and this struct is the row, so a 343 MB pack's rows
228/// cost the length of the pack's entry list rather than the length of its
229/// history.
230#[derive(Debug, Clone)]
231pub struct Resolved {
232 pub oid: Vec<u8>,
233 /// Byte extent of the **stored, verbatim** entry.
234 pub offset: u64,
235 pub len: u64,
236 /// The type the entry carries in the pack — which for a delta is
237 /// `OfsDelta` / `RefDelta`, exactly as `.idx` cannot tell you and this index
238 /// exists to record.
239 pub stored_type: ObjType,
240 /// The type the chain resolves to. Always one of the four real git types.
241 pub kind: GitObjectKind,
242 /// Inflated, **post-delta-resolution** size — the fact `.idx` and `.rev`
243 /// together still cannot answer.
244 pub uncompressed_size: u64,
245 /// The `objects.delta_base` column: an archive **offset**, `0` for none.
246 pub delta_base: u64,
247}
248
249impl Resolved {
250 /// The row [`crate::read_stack::ObjectReadStack::append`] takes.
251 ///
252 /// `delta_base` now has a column of its own in [`IndexEntry`] and rides
253 /// across in it — it was carried on this struct alone until that column
254 /// existed, and was deliberately never smuggled into `uncompressed_size` or
255 /// any other field on the way. **This is the only place it is written into
256 /// an index row** (LAW 5): [`resolve_walked`] computes it once, in the
257 /// archive's coordinate space, and nothing downstream recomputes it from the
258 /// walk.
259 pub fn index_entry(&self) -> IndexEntry {
260 IndexEntry {
261 oid: self.oid.clone(),
262 offset: self.offset,
263 len: self.len,
264 obj_type: self.stored_type,
265 uncompressed_size: self.uncompressed_size,
266 delta_base: self.delta_base,
267 }
268 }
269}
270
271/// The one failure that is a *decision* rather than a corrupt pack.
272#[derive(Debug)]
273pub struct ResolveError;
274
275impl ResolveError {
276 /// A thin pack's external base, named.
277 pub fn missing_base(oid: &[u8]) -> anyhow::Error {
278 anyhow!(
279 "this pack is thin: it deltas against {} which is not in it, and no base source could \
280 produce that object's content — §14's exploded table does not hold it and it could \
281 not be re-derived from the verbatim packs either, so this repository genuinely does \
282 not have that object. The pack is refused by name rather than half-resolved",
283 hex::encode(oid)
284 )
285 }
286
287 /// The worker was told to stop (the repository is being deleted).
288 pub fn cancelled() -> anyhow::Error {
289 anyhow!("index work cancelled: the repository is being deleted")
290 }
291
292 /// **This box cannot index this pack**, said out loud instead of waited out.
293 ///
294 /// The memory watchdog paused, dropped everything droppable, and still could
295 /// not get back over its floor — either for
296 /// [`MemWatchdog::pause_deadline`](crate::memguard::MemWatchdog::pause_deadline)
297 /// or because the flush freed nothing three times running. Lane P2's rig
298 /// took the other branch, which was to wait forever: 1 pause, 0 resumes,
299 /// 1h45m41s, and a client that never got an ack. A refusal by name is a
300 /// worse outcome for this one push and a far better one for the operator,
301 /// who can now read the row and give the box more memory.
302 pub fn out_of_memory(reason: &str, available: u64, budget: u64, done: u64, total: u64) -> anyhow::Error {
303 anyhow!(
304 "this box cannot index this pack: the memory watchdog gave up ({reason}) with {} \
305 available against a {} floor, after {done} of {total} objects. Everything droppable \
306 was dropped and the floor was still breached — what holds the memory is the \
307 resolver's own per-object state, which is proportional to the pack. Give the box \
308 more RAM, lower ZNIPPY_GIT_MEM_BUDGET_PCT, or push a smaller pack. The pack's bytes \
309 are durable and untouched; nothing partial was indexed.",
310 crate::memguard::fmt_bytes(available),
311 crate::memguard::fmt_bytes(budget)
312 )
313 }
314}
315
316/// Default share of `MemTotal` the held base set may take.
317pub const DEFAULT_BASE_BUDGET_PCT: u64 = 25;
318/// Default rows per batch handed to the row sink.
319pub const DEFAULT_ROW_BATCH: usize = 262_144;
320/// **Ceiling on how far the workers may run ahead of the ordered stream**, in
321/// bytes of rows and of the payloads the sink asked to be given. Past it a
322/// worker waits for the emitter instead of buffering the pack — see
323/// [`ResolveLimits::lookahead_bytes`].
324pub const DEFAULT_LOOKAHEAD_BYTES: u64 = 32 << 20;
325/// A ceiling on the worker count no environment variable may cross. A pack is
326/// resolved by the box, not by a number somebody typed.
327pub const MAX_WORKERS: usize = 256;
328
329/// The knobs and hooks the indexer hands the resolver. [`ResolveLimits::none`]
330/// is the old behaviour: unbounded bases, no watchdog, no progress.
331#[derive(Clone)]
332pub struct ResolveLimits<'a> {
333 /// Ceiling on held delta-base bytes; `0` = unbounded.
334 pub base_budget: u64,
335 /// Rows per batch to the row sink.
336 pub row_batch: usize,
337 /// **How many subtrees are walked at once.** `1` is the serial resolver and
338 /// the reference every other value is compared against; the output is the
339 /// same stream at any value (see `the_same_pack_resolves_to_the_same_bytes_at_1_2_and_4_workers`).
340 pub workers: usize,
341 /// Bytes of resolved-but-not-yet-emitted objects the workers may hold. `0`
342 /// takes [`DEFAULT_LOOKAHEAD_BYTES`].
343 pub lookahead_bytes: u64,
344 /// Objects between two checkpoints (cancel flag, watchdog).
345 pub check_every: u64,
346 pub watchdog: Option<Arc<MemWatchdog>>,
347 pub cancel: Option<&'a AtomicBool>,
348 pub progress: Option<&'a IndexProgress>,
349 /// Names the repository in log lines.
350 pub label: &'a str,
351 /// Called once per watchdog pause and again every re-flush: the owner drops
352 /// whatever it holds that can be rebuilt (the store's caches) and returns
353 /// **how much it dropped**. Zero means the owner had nothing to give, which
354 /// is the fact `index.spilled` reported 627 times and nobody acted on —
355 /// see [`crate::memguard::PauseEnd::NothingToGive`].
356 pub on_pause: Option<&'a (dyn Fn() -> u64 + Sync)>,
357}
358
359impl<'a> ResolveLimits<'a> {
360 /// Unbounded, unwatched, unreported — what a test or a one-off resolve uses.
361 pub fn none() -> Self {
362 Self {
363 base_budget: 0,
364 row_batch: DEFAULT_ROW_BATCH,
365 workers: 1,
366 lookahead_bytes: DEFAULT_LOOKAHEAD_BYTES,
367 check_every: crate::memguard::CHECK_EVERY_OBJECTS,
368 watchdog: None,
369 cancel: None,
370 progress: None,
371 label: "",
372 on_pause: None,
373 }
374 }
375
376 /// The indexer's: budget and batch from the environment (once per
377 /// process), the process-wide watchdog, this store's cancel flag and
378 /// progress counter.
379 pub fn for_indexer(
380 label: &'a str,
381 cancel: &'a AtomicBool,
382 progress: &'a IndexProgress,
383 watchdog: Arc<MemWatchdog>,
384 on_pause: &'a (dyn Fn() -> u64 + Sync),
385 ) -> Self {
386 Self {
387 base_budget: base_budget_from_env(),
388 row_batch: row_batch_from_env(),
389 workers: workers_from_env(),
390 lookahead_bytes: lookahead_from_env(),
391 check_every: crate::memguard::CHECK_EVERY_OBJECTS,
392 watchdog: Some(watchdog),
393 cancel: Some(cancel),
394 progress: Some(progress),
395 label,
396 on_pause: Some(on_pause),
397 }
398 }
399}
400
401/// `ZNIPPY_GIT_BASE_BUDGET_BYTES`, else 25 % of `MemTotal`, else 1 GiB when
402/// the box cannot be measured. Read once per process.
403pub fn base_budget_from_env() -> u64 {
404 use std::sync::OnceLock;
405 static B: OnceLock<u64> = OnceLock::new();
406 *B.get_or_init(|| {
407 if let Some(b) = crate::arms::read_env(crate::arms::ENV_BASE_BUDGET_BYTES)
408 .and_then(|v| v.parse::<u64>().ok())
409 {
410 return b;
411 }
412 let total = crate::memguard::mem_total();
413 if total == 0 { 1 << 30 } else { total / 100 * DEFAULT_BASE_BUDGET_PCT }
414 })
415}
416
417/// `ZNIPPY_GIT_INDEX_BATCH_ROWS`, else [`DEFAULT_ROW_BATCH`]. Read once per process.
418pub fn row_batch_from_env() -> usize {
419 use std::sync::OnceLock;
420 static B: OnceLock<usize> = OnceLock::new();
421 *B.get_or_init(|| {
422 crate::arms::read_env(crate::arms::ENV_INDEX_BATCH_ROWS)
423 .and_then(|v| v.parse::<usize>().ok())
424 .filter(|n| *n > 0)
425 .unwrap_or(DEFAULT_ROW_BATCH)
426 })
427}
428
429/// `ZNIPPY_GIT_RESOLVE_WORKERS`, else one per core. Read once per process.
430///
431/// **The forest is the parallelism** ([`crate::forest`]): a base and its
432/// dependents are a tree, and two trees share nothing, so N workers is N
433/// independent depth-first walks over one mmapped pack. It is not N copies of
434/// the pack's O(pack) structures — the walk, the forest and the oid table are
435/// read-only and shared — and it is not N base budgets either: the held-base
436/// ceiling is one atomic across all of them (LAW 5).
437pub fn workers_from_env() -> usize {
438 use std::sync::OnceLock;
439 // The ENVIRONMENT is read once (the read-once law); the embedder's default is
440 // live, because a box learns it is a twin AFTER this process started.
441 static W: OnceLock<Option<usize>> = OnceLock::new();
442 let from_env = *W.get_or_init(|| {
443 crate::arms::read_env(crate::arms::ENV_RESOLVE_WORKERS)
444 .and_then(|v| v.parse::<usize>().ok())
445 .filter(|n| *n > 0)
446 });
447 from_env
448 .unwrap_or_else(|| match RESOLVE_WORKERS_DEFAULT.load(Ordering::Relaxed) {
449 0 => std::thread::available_parallelism().map(|x| x.get()).unwrap_or(1),
450 n => n,
451 })
452 .clamp(1, MAX_WORKERS)
453}
454
455/// What [`workers_from_env`] answers when [`crate::arms::ENV_RESOLVE_WORKERS`] is unset.
456/// `0` is "one per core" — the shipped default — and not a worker count, because
457/// [`workers_from_env`] clamps to at least one and a zero-wide resolver is no resolver.
458static RESOLVE_WORKERS_DEFAULT: AtomicUsize = AtomicUsize::new(0);
459
460/// **Set the process-wide default resolver width** for every resolve started AFTER this call
461/// (an explicit [`crate::arms::ENV_RESOLVE_WORKERS`] still wins). `0` restores one per core.
462/// The twin of [`crate::arms::set_graph_cache_default`], for an embedder that learns its role
463/// at runtime and cannot set the environment — edition 2024 makes `set_var` unsafe, and gunnar
464/// forbids `unsafe`. MEASURED (gunnar `twin_snapshot_memory.rs`, 2026-09-15): one worker per
465/// core, each inflating a whole blob, is +133 MiB of RssAnon for a 512 MiB snapshot install on
466/// a many-core box; two workers with a 16 MiB lookahead is +36 MiB, under gunnar's 64 MiB
467/// twin ceiling.
468pub fn set_resolve_workers_default(workers: usize) {
469 RESOLVE_WORKERS_DEFAULT.store(workers, Ordering::Relaxed);
470}
471
472/// `ZNIPPY_GIT_RESOLVE_LOOKAHEAD_BYTES`, else [`DEFAULT_LOOKAHEAD_BYTES`].
473/// Read once per process.
474pub fn lookahead_from_env() -> u64 {
475 use std::sync::OnceLock;
476 // Read-once environment, live embedder default — see [`workers_from_env`].
477 static L: OnceLock<Option<u64>> = OnceLock::new();
478 let from_env = *L.get_or_init(|| {
479 crate::arms::read_env(crate::arms::ENV_RESOLVE_LOOKAHEAD_BYTES)
480 .and_then(|v| v.parse::<u64>().ok())
481 .filter(|n| *n > 0)
482 });
483 from_env.unwrap_or_else(|| match RESOLVE_LOOKAHEAD_DEFAULT.load(Ordering::Relaxed) {
484 0 => DEFAULT_LOOKAHEAD_BYTES,
485 n => n,
486 })
487}
488
489/// What [`lookahead_from_env`] answers when [`crate::arms::ENV_RESOLVE_LOOKAHEAD_BYTES`] is
490/// unset. `0` is [`DEFAULT_LOOKAHEAD_BYTES`] and not "no lookahead", which is the same
491/// reading the environment door gives a `0` there.
492static RESOLVE_LOOKAHEAD_DEFAULT: AtomicU64 = AtomicU64::new(0);
493
494/// **Set the process-wide default resolver lookahead**, in bytes, for every resolve started
495/// AFTER this call (an explicit [`crate::arms::ENV_RESOLVE_LOOKAHEAD_BYTES`] still wins).
496/// `0` restores [`DEFAULT_LOOKAHEAD_BYTES`]. The twin of
497/// [`crate::arms::set_graph_cache_default`] — see [`set_resolve_workers_default`] for the
498/// measurement the pair is for.
499pub fn set_resolve_lookahead_default(bytes: u64) {
500 RESOLVE_LOOKAHEAD_DEFAULT.store(bytes, Ordering::Relaxed);
501}
502
503/// What one resolve did — the audit line's numbers, all applied output.
504#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
505pub struct ResolveReport {
506 pub rows: u64,
507 /// Bases dropped for the ceiling and inflated again for a later dependent.
508 pub reinflations: u64,
509 /// Bases dropped for the ceiling (each may cost 0..n re-inflations).
510 pub evictions: u64,
511 /// Bases dropped because the watchdog said the box was short.
512 pub watchdog_drops: u64,
513 /// Most bytes of bases held at once.
514 pub peak_base_bytes: u64,
515 /// Watchdog pauses inside this pass.
516 pub pauses: u64,
517 pub paused_ms: u64,
518 /// Row batches handed to the sink.
519 pub batches: u64,
520 /// Times a worker waited for the ordered stream instead of running further
521 /// ahead of it. A positive number is the lookahead ceiling doing its job;
522 /// a large one beside a poor speed-up is the ceiling being the bottleneck.
523 pub stalls: u64,
524 /// How many subtrees were walked at once. `1` is the serial resolver.
525 pub workers: u32,
526 /// Times a worker gave its core back because the box was short of memory
527 /// rather than pausing the whole pack. See `Round::checkpoint`.
528 pub shed: u64,
529 /// The watchdog gave up rather than resumed — the pass is being failed by
530 /// name. See [`ResolveError::out_of_memory`].
531 pub gave_up: bool,
532}
533
534impl ResolveReport {
535 /// Fold one round's numbers into the pack's. Sums, and the two extremes
536 /// taken as extremes — a merge that does not depend on which worker
537 /// produced what, so the audit line is the same at any worker count.
538 pub fn absorb(&mut self, r: ResolveReport) {
539 self.rows += r.rows;
540 self.reinflations += r.reinflations;
541 self.evictions += r.evictions;
542 self.watchdog_drops += r.watchdog_drops;
543 self.peak_base_bytes = self.peak_base_bytes.max(r.peak_base_bytes);
544 self.pauses += r.pauses;
545 self.paused_ms += r.paused_ms;
546 self.batches += r.batches;
547 self.stalls += r.stalls;
548 self.shed += r.shed;
549 self.workers = self.workers.max(r.workers);
550 self.gave_up |= r.gave_up;
551 }
552}
553
554/// Resolve every entry of `pack` to an oid, **keeping no payloads**.
555///
556/// `archive_offset` is where the pack's first byte lives in the archive, so the
557/// offsets and `delta_base`s that come back are already in the archive's
558/// coordinate space and no caller has to remember to shift them.
559///
560/// The indexer does **not** come through here — it calls
561/// [`resolve_streaming`] with the exploded table as its sink, which is what
562/// makes §14's side table fall out of the same pass rather than out of a second
563/// walk.
564pub fn resolve(
565 pack: &[u8],
566 hash: GitHashKind,
567 archive_offset: u64,
568 bases: &dyn BaseSource,
569) -> Result<Vec<Resolved>> {
570 let w = walk(pack, hash.oid_len())?;
571 resolve_walked(pack, &w, hash, archive_offset, bases, &NoSink)
572}
573
574/// Same, for a caller that already walked the pack (`put` does, for the closure
575/// check) and must not walk it twice — and for the one that wants the payloads.
576///
577/// `sink` is handed **every** object exactly once, in resolution order, with the
578/// payload its chain resolves to. Pass [`NoSink`] to keep nothing. This is the
579/// one pass §14's exploded table is built from: no second walk, no second
580/// inflate, no second resolver (LAW 5).
581///
582/// Rows come back in **entry order**. Unbounded bases (the old behaviour): a
583/// caller that wants the ceiling, the watchdog and streaming rows is the
584/// indexer, and it uses [`resolve_streaming`].
585pub fn resolve_walked(
586 pack: &[u8],
587 w: &PackWalk,
588 hash: GitHashKind,
589 archive_offset: u64,
590 bases: &dyn BaseSource,
591 sink: &dyn PayloadSink,
592) -> Result<Vec<Resolved>> {
593 let mut out: Vec<Resolved> = Vec::with_capacity(w.entries.len());
594 let limits = ResolveLimits {
595 row_batch: w.entries.len().max(1),
596 ..ResolveLimits::none()
597 };
598 resolve_streaming(pack, w, hash, archive_offset, bases, sink, limits, &mut |rows| {
599 out.extend(rows.drain(..));
600 Ok(())
601 })?;
602 // Resolution order is entry order except for the ref-deltas a later round
603 // picked up; offsets are unique and ascending in entry order, so this is
604 // exactly the order the old `out[i]` array had.
605 out.sort_unstable_by_key(|r| r.offset);
606 Ok(out)
607}
608
609/// **The resolver**: bounded bases, watchdog checkpoints, rows in batches.
610///
611/// `rows` is called with each full batch (and the final partial one) and may
612/// drain the vector; the same allocation is reused for the next batch. Nothing
613/// about the rows or the sink calls depends on the batch size — it is purely
614/// how much the caller holds at once.
615#[allow(clippy::too_many_arguments)]
616pub fn resolve_streaming(
617 pack: &[u8],
618 w: &PackWalk,
619 hash: GitHashKind,
620 archive_offset: u64,
621 bases: &dyn BaseSource,
622 sink: &dyn PayloadSink,
623 limits: ResolveLimits<'_>,
624 rows: &mut (dyn FnMut(&mut Vec<Resolved>) -> Result<()> + Send),
625) -> Result<ResolveReport> {
626 let mut r = Resolver::new(pack, w, hash, archive_offset, bases, sink, limits)?;
627 r.run(rows)?;
628 Ok(r.report)
629}
630
631
632// ════════════════════════════════════════════════════════════════════════════
633// THE FOREST DRIVER — one subtree per unit, N workers, ONE ordered stream out
634// ════════════════════════════════════════════════════════════════════════════
635
636/// How many objects one worker resolves before it hands them to the ordered
637/// emitter: one lock per few thousand objects instead of one per object.
638const HANDOVER_ROWS: usize = 4096;
639/// …or this many bytes of them, whichever comes first.
640const HANDOVER_BYTES: u64 = 1 << 20;
641/// **How large a scratch buffer may stay between objects.**
642///
643/// `delta` and `leaf` are reused for the whole pass, which is what makes the
644/// hot path allocate nothing — and a `Vec` never gives capacity back, so one
645/// 90 MB blob leaves a 90 MB buffer standing for the remaining thirteen
646/// million objects. Single-threaded that was one such buffer; with a worker per
647/// core it is `2 x workers` of them, and on a 16 GB appliance that is the
648/// difference between finishing a kernel mirror and pausing at 98 %.
649const SCRATCH_KEEP_BYTES: usize = 1 << 20;
650/// What one small heap block costs beyond the bytes asked for — header plus
651/// size-class rounding. Deliberately generous: the lookahead ceiling is a
652/// memory bound, and a memory bound that under-counts is not one.
653const ALLOC_OVERHEAD: usize = 32;
654/// How many objects one pass of the drain takes out of the emit buffer before
655/// it lets the writers back in — the lock-hold bound.
656const DRAIN_RUN: usize = 8192;
657
658/// One frame of a worker's depth-first path: a resolved base, held **only**
659/// while its own subtree is still being walked.
660struct Frame {
661 entry: u32,
662 kind: GitObjectKind,
663 /// Empty when `evicted`.
664 payload: Vec<u8>,
665 evicted: bool,
666 next_child: u32,
667}
668
669/// One resolved object on its way out: the row always, the payload only when
670/// the sink said it would read one ([`PayloadSink::wants`]).
671struct Emit {
672 kind: GitObjectKind,
673 row: Resolved,
674 payload: Option<Vec<u8>>,
675}
676
677impl Emit {
678 /// What holding this costs, for the lookahead ceiling. A count of objects
679 /// cannot tell one commit from one 40 MB blob, so the ceiling is in bytes —
680 /// and the bytes are the ones the ALLOCATOR charges, not the ones the
681 /// struct declares. `ALLOC_OVERHEAD` is the rest of what a 20-byte oid
682 /// costs: the header and the size-class rounding of its own heap block,
683 /// plus this row's share of the `Vec` it sits in. Undercounting it is how a
684 /// 128 MiB ceiling holds half a gigabyte.
685 fn weight(&self) -> u64 {
686 (std::mem::size_of::<Emit>()
687 + ALLOC_OVERHEAD
688 + self.row.oid.len()
689 + self.payload.as_ref().map_or(0, |p| p.len() + ALLOC_OVERHEAD)) as u64
690 }
691}
692
693/// A worker's own scratch: never shared, never locked, reused across the
694/// subtrees one thread walks.
695#[derive(Default)]
696struct Scratch {
697 /// Which unit (root) of this round the worker is inside — the key its
698 /// output is handed over under.
699 unit: usize,
700 delta: Vec<u8>,
701 leaf: Vec<u8>,
702 stack: Vec<Frame>,
703 out: Vec<Emit>,
704 out_bytes: u64,
705 found_refs: Vec<(Vec<u8>, u64)>,
706}
707
708std::thread_local! {
709 /// Per **thread**, not per subtree: a kernel-sized pack has 2.7 M roots and
710 /// allocating two grown buffers for each of them is 2.7 M mallocs the walk
711 /// does not need.
712 static SCRATCH: std::cell::RefCell<Scratch> = std::cell::RefCell::new(Scratch::default());
713}
714
715/// What one root has produced and the drain has not yet handed on, in
716/// depth-first order, and whether the walk of that root is finished.
717///
718/// **One `Vec`, not a queue of them.** The median subtree in a real pack is
719/// ONE object (`examples/forest_stats` on torvalds/linux: p50 1, p90 5), so a
720/// `VecDeque<Vec<Emit>>` per root was two heap allocations and a deque header
721/// to carry a single 20-byte oid — the per-root bookkeeping cost more than the
722/// object it carried, 2.7 M times.
723#[derive(Default)]
724struct RootOut {
725 rows: Vec<Emit>,
726 done: bool,
727}
728
729/// The emit buffer, keyed by **root index**.
730///
731/// # 🔴 It was keyed by ENTRY index, and on a real pack that was a serialiser
732///
733/// Emitting in entry order is the obvious way to make N workers produce the
734/// stream one worker would have: the objects come out sorted by where they sit
735/// in the pack, whoever resolved them. It is also unusable, and the pack says
736/// why. MEASURED on `torvalds/linux` (`examples/forest_stats`), 13 948 244
737/// entries, 2 732 372 roots:
738///
739/// | | subtree **size** | subtree **index span** |
740/// |---|---:|---:|
741/// | p50 | 1 | 0 |
742/// | p90 | 5 | **5 251 404** |
743/// | p99 | 70 | **11 511 634** |
744/// | max | 11 789 | **12 434 450** |
745///
746/// The subtrees are tiny — perfect parallel work — but a five-object subtree
747/// routinely **spans five million entries**, because `pack-objects` puts a base
748/// near its first dependent and the rest of the dependents wherever they fall.
749/// So an entry-ordered cursor is pinned by any subtree in flight whose span
750/// reaches the top of the pack: every other worker's output buffers behind it,
751/// the lookahead ceiling closes, and the workers wait. MEASURED, the kernel
752/// pack at the appliance's 4-core shape: **1 worker 1454.7 s, 2 workers
753/// 1572.9 s** — *slower*, at 1.24 cores of CPU, with 286 686 stalls.
754///
755/// Keyed by root, the cursor moves at the rate roots finish, and roots are
756/// claimed in ascending order, so it follows the workers instead of pinning
757/// them. The order is still fixed and still worker-count independent: roots
758/// ascending, and inside a root the depth-first sequence the children lists
759/// (ascending) determine.
760#[derive(Default)]
761struct Pending {
762 map: std::collections::BTreeMap<usize, RootOut>,
763 bytes: u64,
764 /// **The units a worker is inside right now.** The lookahead ceiling needs
765 /// it: gatling claims a unit *before* the worker runs, so "wait until the
766 /// stream catches up" would otherwise be said by the very worker the stream
767 /// is waiting for. See [`Round::backpressure`].
768 in_flight: std::collections::BTreeSet<usize>,
769 /// How many of those units belong to a worker that has **given its core
770 /// back** for the memory watchdog and is waiting rather than resolving.
771 /// `in_flight.len() - parked` is therefore how many cores this pack is
772 /// actually using, which is the number the last worker standing needs
773 /// before it decides that pausing is the only option left.
774 parked: usize,
775}
776
777/// The ordered end of the pipe. One worker at a time holds it, so the sink and
778/// the row callback see exactly the sequence one worker would have made.
779/// `cursor` is an index into this round's **roots**.
780struct Drain<'r> {
781 cursor: usize,
782 batch: Vec<Resolved>,
783 rows: &'r mut (dyn FnMut(&mut Vec<Resolved>) -> Result<()> + Send),
784}
785
786/// A base this round was promised from outside the pack, probed **once** while
787/// the round was planned so that which entries the round covers is decided
788/// before a byte is inflated.
789struct RoundBases<'r, 'w, 'a> {
790 external: &'r HashMap<&'w [u8], (GitObjectKind, Vec<u8>)>,
791 inner: &'a dyn BaseSource,
792}
793
794impl BaseSource for RoundBases<'_, '_, '_> {
795 fn content(&self, oid: &[u8]) -> Result<Option<(GitObjectKind, Vec<u8>)>> {
796 if let Some((k, p)) = self.external.get(oid) {
797 return Ok(Some((*k, p.clone())));
798 }
799 self.inner.content(oid)
800 }
801}
802
803/// **One round of the fixpoint, across N workers.**
804///
805/// Everything in here is either immutable for the length of the round or an
806/// atomic/lock, because a `&Round` is what every worker holds.
807struct Round<'p, 'w, 'a, 'r> {
808 pack: &'p [u8],
809 w: &'w PackWalk,
810 forest: &'r DeltaForest,
811 hash: GitHashKind,
812 oid_len: usize,
813 archive_offset: u64,
814 bases: &'a dyn BaseSource,
815 sink: &'a dyn PayloadSink,
816 limits: &'r ResolveLimits<'a>,
817 /// The oids some `REF_DELTA` in this pack names. Read-only here.
818 ref_wanted: &'r std::collections::HashSet<&'w [u8]>,
819 /// Where a named oid turned out to live, from the rounds **before** this one.
820 oid_to_offset: &'r HashMap<Vec<u8>, u64>,
821 external: &'r HashMap<&'w [u8], (GitObjectKind, Vec<u8>)>,
822 roots: &'r [u32],
823 in_round: &'r EntrySet,
824 workers: usize,
825 lookahead: u64,
826
827 pend: std::sync::Mutex<Pending>,
828 cv: std::sync::Condvar,
829 drain: std::sync::Mutex<Drain<'r>>,
830 pause_gate: std::sync::Mutex<()>,
831 stop: AtomicBool,
832 err: std::sync::Mutex<Option<(usize, anyhow::Error)>>,
833 since_check: std::sync::atomic::AtomicU64,
834
835 rows: std::sync::atomic::AtomicU64,
836 reinflations: std::sync::atomic::AtomicU64,
837 evictions: std::sync::atomic::AtomicU64,
838 watchdog_drops: std::sync::atomic::AtomicU64,
839 /// **The whole job's** held base bytes, across every worker — lane Q's one
840 /// budget, not one per thread (LAW 5).
841 live_path: std::sync::atomic::AtomicU64,
842 peak_path: std::sync::atomic::AtomicU64,
843 pauses: std::sync::atomic::AtomicU64,
844 paused_ms: std::sync::atomic::AtomicU64,
845 batches: std::sync::atomic::AtomicU64,
846 stalls: std::sync::atomic::AtomicU64,
847 shed: std::sync::atomic::AtomicU64,
848 gave_up: AtomicBool,
849 found_refs: std::sync::Mutex<Vec<(Vec<u8>, u64)>>,
850}
851
852impl<'p, 'w, 'a, 'r> Round<'p, 'w, 'a, 'r> {
853 // ── one subtree ─────────────────────────────────────────────────────────
854
855 /// The unit of work: one root, its whole subtree, in a fixed depth-first
856 /// order.
857 fn run_root(&self, ri: usize, s: &mut Scratch) -> Result<()> {
858 // A previous unit that failed may have left a path standing; nothing
859 // else can, because a finished walk pops every frame.
860 if !s.stack.is_empty() {
861 self.drop_path(s);
862 s.stack.clear();
863 }
864 s.unit = ri;
865 let root = self.roots[ri];
866 let r = (|| {
867 let (kind, payload) = self.root_content(root)?;
868 self.emit(root, kind, &payload, s)?;
869 if self.forest.children(root).is_empty() {
870 drop(payload);
871 } else {
872 self.push_frame(root, kind, payload, s);
873 self.walk(s)?;
874 }
875 Ok(())
876 })();
877 // **The unit is closed even when it failed**, so the drain is never
878 // left waiting on a root that will never say it is finished.
879 self.flush_scratch(s, true)?;
880 r
881 }
882
883 /// A root's own content. `DeltaBase::None` inflates; a `REF_DELTA` root
884 /// applies its delta to a base this round already promised — either an
885 /// entry a previous round placed, or one the [`BaseSource`] answered for.
886 fn root_content(&self, root: u32) -> Result<(GitObjectKind, Vec<u8>)> {
887 let e = &self.w.entries[root as usize];
888 match &e.delta_base {
889 DeltaBase::None => {
890 let kind = whole_kind(e.obj_type)?;
891 let mut out = Vec::with_capacity(e.uncompressed_size as usize);
892 inflate_into(self.pack, e, self.oid_len, &mut out)?;
893 Ok((kind, out))
894 }
895 DeltaBase::Ref(oid) => {
896 let (kind, base) = match self.oid_to_offset.get(oid.as_slice()).copied() {
897 Some(o) => {
898 let be = self
899 .w
900 .entries
901 .binary_search_by_key(&o, |x| x.offset)
902 .map_err(|_| {
903 anyhow!("no entry starts at offset {o} — the pack is corrupt")
904 })?;
905 self.materialize(be as u32)?
906 }
907 None => match self.external.get(oid.as_slice()) {
908 Some((k, p)) => (*k, p.clone()),
909 None => return Err(ResolveError::missing_base(oid)),
910 },
911 };
912 let mut delta = Vec::with_capacity(e.uncompressed_size as usize);
913 inflate_into(self.pack, e, self.oid_len, &mut delta)?;
914 let mut out = Vec::new();
915 apply_delta_into(&base, &delta, &mut out)?;
916 Ok((kind, out))
917 }
918 DeltaBase::Offset(_) => bail!(
919 "the entry at offset {} deltas against an offset, so it is not a root of a \
920 resolve round — the forest is inconsistent with the walk",
921 e.offset
922 ),
923 }
924 }
925
926 /// **The walk.** An explicit stack, not recursion: a verbatim archive grows
927 /// one delta link per push ([`MAX_DELTA_LINKS`]), so a legitimate chain has
928 /// no depth this code may assume.
929 fn walk(&self, s: &mut Scratch) -> Result<()> {
930 while !s.stack.is_empty() {
931 let fi = s.stack.len() - 1;
932 let entry = s.stack[fi].entry;
933 let nkids = self.forest.children(entry).len();
934 let nc = s.stack[fi].next_child as usize;
935 if nc == nkids {
936 self.pop_frame(s);
937 continue;
938 }
939 let c = self.forest.children(entry)[nc];
940 s.stack[fi].next_child += 1;
941 let last = nc + 1 == nkids;
942 let kind = s.stack[fi].kind;
943
944 // The base for this child: moved out on the last one, cloned
945 // otherwise, re-inflated if the ceiling took it.
946 let base: Vec<u8> = if s.stack[fi].evicted {
947 let (_, p) = self.materialize(entry)?;
948 self.reinflations.fetch_add(1, Ordering::Relaxed);
949 if let Some(pr) = self.limits.progress {
950 pr.add_reinflations(1);
951 }
952 if !last {
953 let n = p.len() as u64;
954 let budget = self.limits.base_budget;
955 if budget == 0 || self.live_path.load(Ordering::Relaxed) + n <= budget {
956 // Room for it now: keep the re-inflated copy for the
957 // next child rather than paying again. No eviction to
958 // make room — that would thrash.
959 s.stack[fi].payload = p.clone();
960 s.stack[fi].evicted = false;
961 let live = self.live_path.fetch_add(n, Ordering::Relaxed) + n;
962 self.peak_path.fetch_max(live, Ordering::Relaxed);
963 }
964 }
965 p
966 } else if last {
967 let p = std::mem::take(&mut s.stack[fi].payload);
968 self.live_path
969 .fetch_sub(p.len() as u64, Ordering::Relaxed);
970 s.stack[fi].evicted = true;
971 p
972 } else {
973 s.stack[fi].payload.clone()
974 };
975
976 let ce = &self.w.entries[c as usize];
977 // The scratch buffers are taken out of `s` for the length of the
978 // call and put back on **every** path, error included: a bail that
979 // leaves them behind would make the next child allocate again.
980 let mut delta = std::mem::take(&mut s.delta);
981 let mut leaf = std::mem::take(&mut s.leaf);
982 let leafy = self.forest.children(c).is_empty();
983 let mut carried: Option<Vec<u8>> = None;
984 let r = (|| -> Result<()> {
985 inflate_into(self.pack, ce, self.oid_len, &mut delta)?;
986 if leafy {
987 // A leaf's payload outlives nothing: it is applied into the
988 // reused buffer, hashed in place and handed over from there.
989 apply_delta_into(&base, &delta, &mut leaf)?;
990 drop(base);
991 self.emit(c, kind, &leaf, s)
992 } else {
993 let mut out = Vec::with_capacity(ce.uncompressed_size as usize);
994 apply_delta_into(&base, &delta, &mut out)?;
995 drop(base);
996 self.emit(c, kind, &out, s)?;
997 carried = Some(out);
998 Ok(())
999 }
1000 })();
1001 // Give back what one big object borrowed: the buffers stay warm at
1002 // `SCRATCH_KEEP_BYTES` and never carry the largest object in the
1003 // pack for the rest of it.
1004 delta.shrink_to(SCRATCH_KEEP_BYTES);
1005 leaf.shrink_to(SCRATCH_KEEP_BYTES);
1006 s.delta = delta;
1007 s.leaf = leaf;
1008 r?;
1009 if let Some(out) = carried {
1010 self.push_frame(c, kind, out, s);
1011 }
1012 }
1013 Ok(())
1014 }
1015
1016 fn push_frame(&self, entry: u32, kind: GitObjectKind, payload: Vec<u8>, s: &mut Scratch) {
1017 let len = payload.len() as u64;
1018 s.stack.push(Frame {
1019 entry,
1020 kind,
1021 payload,
1022 evicted: false,
1023 next_child: 0,
1024 });
1025 self.live_path.fetch_add(len, Ordering::Relaxed);
1026 // **The ceiling first, the high-water mark after.** Recording the peak
1027 // before the eviction counted the base that was about to be dropped,
1028 // so a one-byte budget reported two live bases and the report said the
1029 // ceiling had leaked when it had not.
1030 self.enforce_budget(s);
1031 self.peak_path
1032 .fetch_max(self.live_path.load(Ordering::Relaxed), Ordering::Relaxed);
1033 }
1034
1035 fn pop_frame(&self, s: &mut Scratch) {
1036 if let Some(f) = s.stack.pop()
1037 && !f.evicted
1038 {
1039 self.live_path
1040 .fetch_sub(f.payload.len() as u64, Ordering::Relaxed);
1041 }
1042 }
1043
1044 /// **Belady on a depth-first path.** The walk returns to the shallowest
1045 /// held frame only after every deeper subtree is finished, so the
1046 /// shallowest is by construction the one whose next use is farthest away —
1047 /// which is exactly the base the ceiling should drop. No `next_use` table
1048 /// is needed to know it; the shape says so.
1049 fn enforce_budget(&self, s: &mut Scratch) {
1050 let budget = self.limits.base_budget;
1051 if budget == 0 {
1052 return;
1053 }
1054 let mut i = 0usize;
1055 // The deepest frame is the one the next child reads from, so it is
1056 // never the eviction candidate — dropping it would re-inflate it on the
1057 // very next line.
1058 while i + 1 < s.stack.len() && self.live_path.load(Ordering::Relaxed) > budget {
1059 if !s.stack[i].evicted {
1060 let n = s.stack[i].payload.len() as u64;
1061 s.stack[i].payload = Vec::new();
1062 s.stack[i].evicted = true;
1063 self.live_path.fetch_sub(n, Ordering::Relaxed);
1064 self.evictions.fetch_add(1, Ordering::Relaxed);
1065 }
1066 i += 1;
1067 }
1068 }
1069
1070 /// Every payload on this worker's path, dropped, and **how many bytes that
1071 /// freed** — the number the watchdog compares with zero.
1072 fn drop_path(&self, s: &mut Scratch) -> u64 {
1073 let mut freed = 0u64;
1074 for f in s.stack.iter_mut() {
1075 if !f.evicted {
1076 freed += f.payload.len() as u64;
1077 f.payload = Vec::new();
1078 f.evicted = true;
1079 }
1080 }
1081 self.live_path.fetch_sub(freed, Ordering::Relaxed);
1082 freed
1083 }
1084
1085 /// Re-inflate an entry from the mmapped pack: its own chain, re-applied.
1086 /// **The one chain walker** (LAW 5) — [`materialize_entry`], with this
1087 /// round's promised external bases in front of the store's.
1088 fn materialize(&self, entry: u32) -> Result<(GitObjectKind, Vec<u8>)> {
1089 let rb = RoundBases {
1090 external: self.external,
1091 inner: self.bases,
1092 };
1093 let of = |oid: &[u8]| self.oid_to_offset.get(oid).copied();
1094 materialize_entry(self.pack, self.w, entry as usize, self.oid_len, &of, &rb)
1095 }
1096
1097 // ── the ordered end ─────────────────────────────────────────────────────
1098
1099 fn emit(&self, index: u32, kind: GitObjectKind, payload: &[u8], s: &mut Scratch) -> Result<()> {
1100 let e = &self.w.entries[index as usize];
1101 let oid = self.hash.oid_of_object(kind, payload);
1102 // Only now is "does a REF_DELTA name this object" answerable, because it
1103 // is keyed by the oid this line just computed.
1104 if self.ref_wanted.contains(&oid.as_slice()) {
1105 s.found_refs.push((oid.clone(), e.offset));
1106 }
1107 let want = self.sink.wants(kind);
1108 let em = Emit {
1109 kind,
1110 row: Resolved {
1111 oid,
1112 offset: e.offset + self.archive_offset,
1113 len: e.len,
1114 stored_type: e.obj_type,
1115 kind,
1116 uncompressed_size: payload.len() as u64,
1117 delta_base: match &e.delta_base {
1118 DeltaBase::Offset(o) => o + self.archive_offset,
1119 _ => 0,
1120 },
1121 },
1122 payload: if want { Some(payload.to_vec()) } else { None },
1123 };
1124 s.out_bytes += em.weight();
1125 s.out.push(em);
1126 self.rows.fetch_add(1, Ordering::Relaxed);
1127 if let Some(p) = self.limits.progress {
1128 p.object_done(e.len);
1129 }
1130 if s.out.len() >= HANDOVER_ROWS || s.out_bytes >= HANDOVER_BYTES {
1131 self.flush_scratch(s, false)?;
1132 }
1133 let n = self.since_check.fetch_add(1, Ordering::Relaxed) + 1;
1134 if n >= self.limits.check_every {
1135 self.since_check.store(0, Ordering::Relaxed);
1136 self.checkpoint(s)?;
1137 }
1138 Ok(())
1139 }
1140
1141 /// Hand this worker's finished objects to the ordered emitter, then try to
1142 /// push the stream forward. Never blocks on the drain: whoever holds it is
1143 /// already doing the work. `done` closes the unit — the drain may then step
1144 /// past it to the next root.
1145 fn flush_scratch(&self, s: &mut Scratch, done: bool) -> Result<()> {
1146 if !s.out.is_empty() || done {
1147 let mut bytes = 0u64;
1148 for em in s.out.iter() {
1149 bytes += em.weight();
1150 }
1151 let unit = s.unit;
1152 {
1153 let mut p = self.lock_pend()?;
1154 let e = p.map.entry(unit).or_default();
1155 if e.rows.is_empty() {
1156 e.rows = std::mem::take(&mut s.out);
1157 } else {
1158 e.rows.append(&mut s.out);
1159 }
1160 e.done |= done;
1161 p.bytes += bytes;
1162 }
1163 s.out.clear();
1164 s.out.shrink_to(HANDOVER_ROWS);
1165 s.out_bytes = 0;
1166 }
1167 self.try_drain(false)
1168 }
1169
1170 fn lock_pend(&self) -> Result<std::sync::MutexGuard<'_, Pending>> {
1171 self.pend
1172 .lock()
1173 .map_err(|_| anyhow!("the resolver's emit buffer is poisoned"))
1174 }
1175
1176 /// **The stream out.** This round's roots in ascending order, and inside
1177 /// each root the depth-first sequence its children lists fix — to the sink
1178 /// first, then to the row callback.
1179 fn try_drain(&self, force: bool) -> Result<()> {
1180 let mut guard = match self.drain.try_lock() {
1181 Ok(g) => g,
1182 Err(std::sync::TryLockError::WouldBlock) => {
1183 if !force {
1184 return Ok(());
1185 }
1186 self.drain
1187 .lock()
1188 .map_err(|_| anyhow!("the resolver's row drain is poisoned"))?
1189 }
1190 Err(std::sync::TryLockError::Poisoned(_)) => {
1191 bail!("the resolver's row drain is poisoned")
1192 }
1193 };
1194 let d = &mut *guard;
1195 loop {
1196 let mut run: Vec<Vec<Emit>> = Vec::new();
1197 let mut held = 0usize;
1198 let mut moved = false;
1199 {
1200 let mut p = self.lock_pend()?;
1201 while d.cursor < self.roots.len() {
1202 let (rows, done) = match p.map.get_mut(&d.cursor) {
1203 Some(ro) => (std::mem::take(&mut ro.rows), ro.done),
1204 None => break,
1205 };
1206 if !rows.is_empty() {
1207 let n: u64 = rows.iter().map(Emit::weight).sum();
1208 p.bytes = p.bytes.saturating_sub(n);
1209 held += rows.len();
1210 run.push(rows);
1211 moved = true;
1212 }
1213 if !done {
1214 break;
1215 }
1216 p.map.remove(&d.cursor);
1217 d.cursor += 1;
1218 moved = true;
1219 if held >= DRAIN_RUN {
1220 break;
1221 }
1222 }
1223 if moved {
1224 self.cv.notify_all();
1225 }
1226 }
1227 if !moved {
1228 return Ok(());
1229 }
1230 for chunk in run {
1231 for em in chunk {
1232 if let Some(pay) = &em.payload {
1233 self.sink.explode(&em.row.oid, em.kind, pay)?;
1234 }
1235 d.batch.push(em.row);
1236 if d.batch.len() >= self.limits.row_batch {
1237 (d.rows)(&mut d.batch)?;
1238 self.batches.fetch_add(1, Ordering::Relaxed);
1239 d.batch.clear();
1240 }
1241 }
1242 }
1243 }
1244 }
1245
1246 /// Everything left, in order, at the end of a round.
1247 fn finish(&self) -> Result<()> {
1248 self.try_drain(true)?;
1249 let mut d = self
1250 .drain
1251 .lock()
1252 .map_err(|_| anyhow!("the resolver's row drain is poisoned"))?;
1253 if !d.batch.is_empty() {
1254 let d = &mut *d;
1255 (d.rows)(&mut d.batch)?;
1256 self.batches.fetch_add(1, Ordering::Relaxed);
1257 d.batch.clear();
1258 }
1259 let p = self.lock_pend()?;
1260 if !p.map.is_empty() || d.cursor != self.roots.len() {
1261 bail!(
1262 "the resolver finished a round at root {} of {} with {} roots still unemitted — \
1263 the round's roots and the walk disagree",
1264 d.cursor,
1265 self.roots.len(),
1266 p.map.len()
1267 );
1268 }
1269 Ok(())
1270 }
1271
1272 /// **The lookahead ceiling, and the one worker it may never stop.**
1273 ///
1274 /// A worker that has run far ahead of the ordered stream waits here rather
1275 /// than buffering the pack. The rule for *who* waits is the whole
1276 /// correctness argument: the worker holding the **lowest unit in flight**
1277 /// never waits. Gatling hands its units out in ascending order, so every
1278 /// lower unit is already finished, and the entry the ordered cursor is
1279 /// stuck on therefore belongs either to a finished unit — already in the
1280 /// buffer, and the drain will take it — or to the lowest unit in flight,
1281 /// which is running. Progress is guaranteed either way.
1282 ///
1283 /// 🔴 It said "one worker is never allowed to wait, whichever it is", and
1284 /// that HUNG: gatling claims a unit *before* it calls the closure, so the
1285 /// worker allowed to run on was frequently not the one the cursor was
1286 /// waiting for. It then consumed every remaining unit while the worker
1287 /// holding the cursor's own unit sat in `wait` forever, and the scope never
1288 /// joined. Caught by
1289 /// `the_same_pack_resolves_to_the_same_bytes_at_1_2_and_4_workers`, which
1290 /// ran for over 60 s and had to be killed.
1291 fn backpressure(&self, ri: usize) -> Result<()> {
1292 self.try_drain(false)?;
1293 let mut p = self.lock_pend()?;
1294 while p.bytes > self.lookahead
1295 && !self.stop.load(Ordering::Relaxed)
1296 && p.in_flight.first().is_some_and(|lowest| *lowest < ri)
1297 {
1298 self.stalls.fetch_add(1, Ordering::Relaxed);
1299 p = self
1300 .cv
1301 .wait(p)
1302 .map_err(|_| anyhow!("the resolver's emit buffer is poisoned"))?;
1303 }
1304 Ok(())
1305 }
1306
1307 /// Enter and leave a unit. The set is what [`backpressure`](Self::backpressure)
1308 /// reads, and leaving one is what lets a waiting worker re-decide.
1309 fn enter(&self, ri: usize) -> Result<()> {
1310 self.lock_pend()?.in_flight.insert(ri);
1311 Ok(())
1312 }
1313
1314 fn leave(&self, ri: usize) {
1315 if let Ok(mut p) = self.pend.lock() {
1316 p.in_flight.remove(&ri);
1317 }
1318 self.cv.notify_all();
1319 }
1320
1321 // ── the checkpoints ─────────────────────────────────────────────────────
1322
1323 /// Every [`crate::memguard::CHECK_EVERY_OBJECTS`] objects **of the whole
1324 /// job**: another worker's failure, the cancel flag, then the watchdog.
1325 fn checkpoint(&self, s: &mut Scratch) -> Result<()> {
1326 if self.stop.load(Ordering::Relaxed) {
1327 bail!("this pack is being failed by another worker");
1328 }
1329 if self.limits.cancel.is_some_and(|c| c.load(Ordering::Relaxed)) {
1330 return Err(ResolveError::cancelled());
1331 }
1332 let Some(dog) = self.limits.watchdog.clone() else {
1333 return Ok(());
1334 };
1335 if dog.breached().is_none() {
1336 return Ok(());
1337 }
1338 // My own share of the problem, dropped before anybody waits.
1339 let mut freed = self.drop_path(s);
1340
1341 // ── GIVE THE CORE BACK BEFORE GIVING THE PACK UP ────────────────────
1342 //
1343 // **A short box wants fewer workers, not a refused pack.** MEASURED on
1344 // the rig, the 16 GB appliance and the 8.12 GiB kernel mirror: one
1345 // worker finishes it with 5.55 GiB to spare, two stop at 96 % and four
1346 // at 86 %. The cost is not the resolver's own structures — peak RSS at
1347 // four workers (13.7 GiB) is BELOW the single-threaded fixpoint's
1348 // (14.04 GiB) — it is that four depth-first cursors over a pack whose
1349 // subtrees span millions of entries keep far more of the 8.12 GiB
1350 // mapping resident at once, and `MemAvailable` counts half of page
1351 // cache.
1352 //
1353 // So a worker that finds the box short **stops being a worker**: it
1354 // waits until it is the lowest unit in flight, which is exactly when
1355 // every worker above it has drained, and the round finishes on one
1356 // core instead of being refused on four. It never deadlocks for the
1357 // same reason [`backpressure`](Self::backpressure) does not — the
1358 // lowest unit in flight never waits, and gatling hands units out in
1359 // ascending order, so the lowest always advances.
1360 //
1361 // The ordered emitter means this changes only WHEN work happens, never
1362 // in what order it comes out: the stream is byte-identical either way.
1363 {
1364 let mut p = self.lock_pend()?;
1365 let mut waited = false;
1366 loop {
1367 if self.stop.load(Ordering::Relaxed) {
1368 break;
1369 }
1370 let lowest = p.in_flight.first().copied();
1371 if lowest.is_some_and(|l| l < s.unit) {
1372 // Somebody below me is still the one the ordered stream is
1373 // waiting for: I am the expendable one. Park.
1374 if !waited {
1375 waited = true;
1376 p.parked += 1;
1377 self.shed.fetch_add(1, Ordering::Relaxed);
1378 if let Some(pr) = self.limits.progress {
1379 pr.add_shed();
1380 pr.set_phase(Phase::Paused);
1381 }
1382 self.cv.notify_all();
1383 }
1384 p = self
1385 .cv
1386 .wait(p)
1387 .map_err(|_| anyhow!("the resolver's emit buffer is poisoned"))?;
1388 continue;
1389 }
1390 // I am the lowest unit in flight — the one that must carry on.
1391 // **Do not pause while other workers are still resolving**:
1392 // they are what the box cannot afford, and each of them will
1393 // park at its own next checkpoint. Pausing here instead would
1394 // stop the only worker that is allowed to make progress and
1395 // then time out on a box that was about to have room.
1396 if p.in_flight.len() > p.parked + 1 {
1397 if waited {
1398 waited = false;
1399 p.parked -= 1;
1400 }
1401 drop(p);
1402 if let Some(pr) = self.limits.progress {
1403 pr.set_phase(Phase::Resolve);
1404 }
1405 return Ok(());
1406 }
1407 break;
1408 }
1409 if waited {
1410 p.parked -= 1;
1411 }
1412 drop(p);
1413 if waited {
1414 if let Some(pr) = self.limits.progress {
1415 pr.set_phase(Phase::Resolve);
1416 }
1417 if self.stop.load(Ordering::Relaxed) {
1418 bail!("this pack is being failed by another worker");
1419 }
1420 // The box may well be fine now that everybody above me has
1421 // stopped. Re-ask before waking the watchdog.
1422 if dog.breached().is_none() {
1423 return Ok(());
1424 }
1425 }
1426 }
1427
1428 if let Some(p) = self.limits.progress {
1429 p.set_phase(Phase::Paused);
1430 p.add_pause();
1431 }
1432 // **Only one worker drives the pause**; the others block on this gate,
1433 // which is the point — an indexer that keeps three threads allocating
1434 // through a pause is not paused. They have already dropped their own
1435 // bases on the way in.
1436 let gate = self
1437 .pause_gate
1438 .lock()
1439 .map_err(|_| anyhow!("the resolver's pause gate is poisoned"))?;
1440 // 🔴 **A give-up is the whole pack's, not one worker's.** Without this
1441 // line the workers queued on the gate each opened a pause of their own
1442 // after the first had already given up. MEASURED on the rig, two
1443 // workers: `gave up after 30681 ms (nothing-to-give)` at 00:59:33, then
1444 // a second pause opened in the same millisecond and ran the FULL
1445 // `PAUSE_DEADLINE` — `gave up after 300164 ms (deadline)` at 01:04:33.
1446 // Five minutes of a client waiting for a verdict the box had already
1447 // reached. One give-up stops the round.
1448 if self.stop.load(Ordering::Relaxed) {
1449 drop(gate);
1450 bail!("this pack is being failed by another worker");
1451 }
1452 if dog.breached().is_none() {
1453 drop(gate);
1454 if let Some(p) = self.limits.progress {
1455 p.set_phase(Phase::Resolve);
1456 }
1457 return Ok(());
1458 }
1459 let (cancel, on_pause, label) = (self.limits.cancel, self.limits.on_pause, self.limits.label);
1460 let mut flush = || -> Result<u64> {
1461 let mut f = std::mem::take(&mut freed);
1462 self.watchdog_drops.fetch_add(1, Ordering::Relaxed);
1463 self.sink.flush()?;
1464 // `false`: the unit is mid-walk, so the drain must not step past
1465 // this root — it is not finished, it is paused.
1466 self.flush_scratch(s, false)?;
1467 self.try_drain(true)?;
1468 if let Some(g) = on_pause {
1469 f = f.saturating_add(g());
1470 }
1471 Ok(f)
1472 };
1473 let paused = dog.pause_until_ok(label, cancel, &mut flush);
1474 let ms = match &paused {
1475 Ok(p) => p.ms,
1476 Err(_) => 0,
1477 };
1478 self.pauses.fetch_add(1, Ordering::Relaxed);
1479 self.paused_ms.fetch_add(ms, Ordering::Relaxed);
1480 if let Some(p) = self.limits.progress {
1481 p.set_phase(Phase::Resolve);
1482 }
1483 drop(gate);
1484 let paused = paused?;
1485 if self.limits.cancel.is_some_and(|c| c.load(Ordering::Relaxed)) {
1486 return Err(ResolveError::cancelled());
1487 }
1488 // **A give-up is not a resume.** The pack is refused by name; its bytes
1489 // stay durable and nothing partial is left indexed.
1490 if paused.end.gave_up() {
1491 self.gave_up.store(true, Ordering::Relaxed);
1492 // Before the error is even built: every worker queued behind the
1493 // gate must find the round already lost.
1494 self.stop.store(true, Ordering::Relaxed);
1495 if self.pend.lock().is_ok() {
1496 self.cv.notify_all();
1497 }
1498 let (done, total) = match self.limits.progress {
1499 Some(p) => {
1500 let sn = p.snapshot();
1501 (sn.objects_done, sn.objects_total)
1502 }
1503 None => (self.rows.load(Ordering::Relaxed), self.w.entries.len() as u64),
1504 };
1505 return Err(ResolveError::out_of_memory(
1506 paused.end.as_str(),
1507 paused.available,
1508 paused.budget,
1509 done,
1510 total,
1511 ));
1512 }
1513 Ok(())
1514 }
1515
1516 /// The first failure by **root index**, so a pack that two workers could
1517 /// fail is refused with the same sentence every time.
1518 fn fail(&self, ri: usize, e: anyhow::Error) {
1519 self.stop.store(true, Ordering::Relaxed);
1520 if let Ok(mut g) = self.err.lock() {
1521 let take = match &*g {
1522 Some((i, _)) => ri < *i,
1523 None => true,
1524 };
1525 if take {
1526 *g = Some((ri, e));
1527 }
1528 }
1529 if self.pend.lock().is_ok() {
1530 self.cv.notify_all();
1531 }
1532 }
1533
1534 fn take_err(&self) -> Option<anyhow::Error> {
1535 self.err.lock().ok().and_then(|mut g| g.take()).map(|(_, e)| e)
1536 }
1537
1538 fn report(&self) -> ResolveReport {
1539 ResolveReport {
1540 rows: self.rows.load(Ordering::Relaxed),
1541 reinflations: self.reinflations.load(Ordering::Relaxed),
1542 evictions: self.evictions.load(Ordering::Relaxed),
1543 watchdog_drops: self.watchdog_drops.load(Ordering::Relaxed),
1544 peak_base_bytes: self.peak_path.load(Ordering::Relaxed),
1545 pauses: self.pauses.load(Ordering::Relaxed),
1546 paused_ms: self.paused_ms.load(Ordering::Relaxed),
1547 batches: self.batches.load(Ordering::Relaxed),
1548 stalls: self.stalls.load(Ordering::Relaxed),
1549 shed: self.shed.load(Ordering::Relaxed),
1550 workers: self.workers as u32,
1551 gave_up: self.gave_up.load(Ordering::Relaxed),
1552 }
1553 }
1554}
1555
1556/// **The resolver.** One walk, one forest, one ordered stream out, and as many
1557/// workers as the box was told to use.
1558struct Resolver<'p, 'w, 'a> {
1559 pack: &'p [u8],
1560 w: &'w PackWalk,
1561 hash: GitHashKind,
1562 oid_len: usize,
1563 archive_offset: u64,
1564 bases: &'a dyn BaseSource,
1565 sink: &'a dyn PayloadSink,
1566 limits: ResolveLimits<'a>,
1567 forest: DeltaForest,
1568 /// The oids `REF_DELTA` entries in this pack name — the half of the
1569 /// dependent set that is not knowable from an offset.
1570 ref_wanted: std::collections::HashSet<&'w [u8]>,
1571 /// …and where they turned out to live, as the rounds place them.
1572 oid_to_offset: HashMap<Vec<u8>, u64>,
1573 report: ResolveReport,
1574}
1575
1576impl<'p, 'w, 'a> Resolver<'p, 'w, 'a> {
1577 fn new(
1578 pack: &'p [u8],
1579 w: &'w PackWalk,
1580 hash: GitHashKind,
1581 archive_offset: u64,
1582 bases: &'a dyn BaseSource,
1583 sink: &'a dyn PayloadSink,
1584 limits: ResolveLimits<'a>,
1585 ) -> Result<Self> {
1586 // **The dependents, as the forest they always were.** Built from
1587 // `PackWalk::entries` and nothing else (LAW 5): `parent`, the CSR child
1588 // lists, and the roots.
1589 let forest = DeltaForest::build(w)?;
1590 let mut ref_wanted: std::collections::HashSet<&'w [u8]> = std::collections::HashSet::new();
1591 for e in w.entries.iter() {
1592 if let DeltaBase::Ref(oid) = &e.delta_base {
1593 ref_wanted.insert(oid.as_slice());
1594 }
1595 }
1596 Ok(Self {
1597 pack,
1598 w,
1599 hash,
1600 oid_len: hash.oid_len(),
1601 archive_offset,
1602 bases,
1603 sink,
1604 forest,
1605 ref_wanted,
1606 oid_to_offset: HashMap::new(),
1607 report: ResolveReport::default(),
1608 limits,
1609 })
1610 }
1611
1612 /// The fixpoint, one round at a time — and each round is a gatling fan-out
1613 /// over its roots.
1614 ///
1615 /// A round is planned before a byte of it is inflated: its roots are the
1616 /// entries whose base is available *now*, and its entry set is the closure
1617 /// of those roots over the forest. That is what lets the ordered emitter
1618 /// know, for every index, whether to wait for it or step past it — and
1619 /// therefore what makes the stream out of N workers the stream out of one.
1620 fn run(&mut self, rows: &mut (dyn FnMut(&mut Vec<Resolved>) -> Result<()> + Send)) -> Result<()> {
1621 let n = self.w.entries.len();
1622 if let Some(p) = self.limits.progress {
1623 p.set_total(n as u64);
1624 p.set_phase(Phase::Resolve);
1625 }
1626 if n == 0 {
1627 return Ok(());
1628 }
1629 let w = self.w;
1630 let mut done = EntrySet::new(n);
1631 let mut round = 0usize;
1632 while done.len() < n {
1633 let mut roots: Vec<u32> = if round == 0 {
1634 self.forest.whole_roots().to_vec()
1635 } else {
1636 Vec::new()
1637 };
1638 // A `REF_DELTA` names a base by oid. It is resolvable this round if
1639 // an earlier round placed that oid inside the pack, or if the store
1640 // can produce it — exactly the two answers the serial fixpoint
1641 // asked for, asked once each, up front, so the round's membership
1642 // is a fact rather than a race.
1643 let mut external: HashMap<&'w [u8], (GitObjectKind, Vec<u8>)> = HashMap::new();
1644 for &r in self.forest.ref_roots() {
1645 if done.contains(r as usize) {
1646 continue;
1647 }
1648 let DeltaBase::Ref(oid) = &w.entries[r as usize].delta_base else {
1649 continue;
1650 };
1651 if self.oid_to_offset.contains_key(oid.as_slice())
1652 || external.contains_key(oid.as_slice())
1653 {
1654 roots.push(r);
1655 continue;
1656 }
1657 if let Some(c) = self.bases.content(oid)? {
1658 external.insert(oid.as_slice(), c);
1659 roots.push(r);
1660 }
1661 }
1662 if roots.is_empty() {
1663 // Nothing can move: every remaining entry waits on a base
1664 // nobody can supply. Name the first one — a thin pack is the
1665 // ordinary reason.
1666 for &r in self.forest.ref_roots() {
1667 if !done.contains(r as usize)
1668 && let DeltaBase::Ref(oid) = &w.entries[r as usize].delta_base
1669 {
1670 return Err(ResolveError::missing_base(oid));
1671 }
1672 }
1673 bail!(
1674 "{} of {n} entries could not be resolved and none of them deltas against an \
1675 oid — the pack is corrupt",
1676 n - done.len()
1677 );
1678 }
1679 roots.sort_unstable();
1680 let in_round = self.forest.closure_of(&roots);
1681 let (rep, found) = self.run_round(&roots, &in_round, &external, rows)?;
1682 self.report.absorb(rep);
1683 for (oid, off) in found {
1684 self.oid_to_offset.insert(oid, off);
1685 }
1686 done.union_with(&in_round);
1687 round += 1;
1688 }
1689 Ok(())
1690 }
1691
1692 /// One round: **the gatling fan-out**. NEVER rayon (LAW 3) — every thread
1693 /// in this constellation originates in `gatling`, and this is its fork-join
1694 /// shape: a fixed `0..roots` of independent units, self-dispatched from one
1695 /// atomic, no barrier between them.
1696 fn run_round(
1697 &self,
1698 roots: &[u32],
1699 in_round: &EntrySet,
1700 external: &HashMap<&'w [u8], (GitObjectKind, Vec<u8>)>,
1701 rows: &mut (dyn FnMut(&mut Vec<Resolved>) -> Result<()> + Send),
1702 ) -> Result<(ResolveReport, Vec<(Vec<u8>, u64)>)> {
1703 let workers = self
1704 .limits
1705 .workers
1706 .max(1)
1707 .min(roots.len().max(1))
1708 .min(MAX_WORKERS);
1709 let round = Round {
1710 pack: self.pack,
1711 w: self.w,
1712 forest: &self.forest,
1713 hash: self.hash,
1714 oid_len: self.oid_len,
1715 archive_offset: self.archive_offset,
1716 bases: self.bases,
1717 sink: self.sink,
1718 limits: &self.limits,
1719 ref_wanted: &self.ref_wanted,
1720 oid_to_offset: &self.oid_to_offset,
1721 external,
1722 roots,
1723 in_round,
1724 workers,
1725 lookahead: self.limits.lookahead_bytes.max(1 << 20),
1726 pend: std::sync::Mutex::new(Pending::default()),
1727 cv: std::sync::Condvar::new(),
1728 drain: std::sync::Mutex::new(Drain {
1729 cursor: 0,
1730 batch: Vec::with_capacity(self.limits.row_batch.min(1 << 20)),
1731 rows,
1732 }),
1733 pause_gate: std::sync::Mutex::new(()),
1734 stop: AtomicBool::new(false),
1735 err: std::sync::Mutex::new(None),
1736 since_check: std::sync::atomic::AtomicU64::new(0),
1737 rows: std::sync::atomic::AtomicU64::new(0),
1738 reinflations: std::sync::atomic::AtomicU64::new(0),
1739 evictions: std::sync::atomic::AtomicU64::new(0),
1740 watchdog_drops: std::sync::atomic::AtomicU64::new(0),
1741 live_path: std::sync::atomic::AtomicU64::new(0),
1742 peak_path: std::sync::atomic::AtomicU64::new(0),
1743 pauses: std::sync::atomic::AtomicU64::new(0),
1744 paused_ms: std::sync::atomic::AtomicU64::new(0),
1745 batches: std::sync::atomic::AtomicU64::new(0),
1746 stalls: std::sync::atomic::AtomicU64::new(0),
1747 shed: std::sync::atomic::AtomicU64::new(0),
1748 gave_up: AtomicBool::new(false),
1749 found_refs: std::sync::Mutex::new(Vec::new()),
1750 };
1751 gatling_for_each_unit(roots.len(), workers, |ri| {
1752 if round.stop.load(Ordering::Relaxed) {
1753 return;
1754 }
1755 let r = SCRATCH.with(|cell| -> Result<()> {
1756 let mut s = cell.borrow_mut();
1757 round.enter(ri)?;
1758 let r = (|| {
1759 round.backpressure(ri)?;
1760 if round.stop.load(Ordering::Relaxed) {
1761 return Ok(());
1762 }
1763 round.run_root(ri, &mut s)
1764 })();
1765 if !s.found_refs.is_empty()
1766 && let Ok(mut g) = round.found_refs.lock()
1767 {
1768 g.append(&mut s.found_refs);
1769 }
1770 round.leave(ri);
1771 r
1772 });
1773 if let Err(e) = r {
1774 round.fail(ri, e);
1775 }
1776 });
1777 if let Some(e) = round.take_err() {
1778 // Everything this round did stays where it is; the caller refuses
1779 // the pack by name and nothing partial is marked indexed.
1780 return Err(e);
1781 }
1782 round.finish()?;
1783 let mut found = round
1784 .found_refs
1785 .lock()
1786 .map_err(|_| anyhow!("the resolver's ref table is poisoned"))?
1787 .split_off(0);
1788 // The order N workers found them in is not an order; sorting makes the
1789 // map the same map at any worker count.
1790 found.sort_unstable();
1791 found.dedup();
1792 Ok((round.report(), found))
1793 }
1794}
1795
1796/// The gatling fork-join, by its own name, with the one-worker case kept off
1797/// the thread pool entirely: `workers == 1` runs the units on the calling
1798/// thread, which is what makes a 1-worker run the reference a 2- and a
1799/// 4-worker run are compared against.
1800fn gatling_for_each_unit<F: Fn(usize) + Sync>(n: usize, workers: usize, f: F) {
1801 znippy_zoomies::gatling_forkjoin::gatling_run(n, workers, f);
1802}
1803
1804
1805/// **One object's content from a pack and an entry, with only its chain
1806/// inflated** — the lazy read path. `offset_of_oid` answers a `REF_DELTA`
1807/// base that is inside the pack (its entry offset); `bases` answers one that is
1808/// not. Nothing is cached: a chain of depth d costs d inflates and d−1 delta
1809/// applications, and nothing else.
1810pub fn materialize_entry(
1811 pack: &[u8],
1812 w: &PackWalk,
1813 entry: usize,
1814 oid_len: usize,
1815 offset_of_oid: &dyn Fn(&[u8]) -> Option<u64>,
1816 bases: &dyn BaseSource,
1817) -> Result<(GitObjectKind, Vec<u8>)> {
1818 let e = w
1819 .entries
1820 .get(entry)
1821 .ok_or_else(|| anyhow!("entry {entry} is past the walk ({} entries)", w.entries.len()))?;
1822 match &e.delta_base {
1823 DeltaBase::None => {
1824 let mut out = Vec::with_capacity(e.uncompressed_size as usize);
1825 inflate_into(pack, e, oid_len, &mut out)?;
1826 Ok((whole_kind(e.obj_type)?, out))
1827 }
1828 DeltaBase::Offset(o) => {
1829 let base_entry = w
1830 .entries
1831 .binary_search_by_key(o, |x| x.offset)
1832 .map_err(|_| anyhow!("no entry starts at offset {o} — the pack is corrupt"))?;
1833 let (kind, base) = materialize_entry(pack, w, base_entry, oid_len, offset_of_oid, bases)?;
1834 let mut delta = Vec::with_capacity(e.uncompressed_size as usize);
1835 inflate_into(pack, e, oid_len, &mut delta)?;
1836 let mut out = Vec::new();
1837 apply_delta_into(&base, &delta, &mut out)?;
1838 Ok((kind, out))
1839 }
1840 DeltaBase::Ref(oid) => {
1841 let (kind, base) = match offset_of_oid(oid) {
1842 Some(o) => {
1843 let base_entry = w
1844 .entries
1845 .binary_search_by_key(&o, |x| x.offset)
1846 .map_err(|_| anyhow!("no entry starts at offset {o} — the pack is corrupt"))?;
1847 materialize_entry(pack, w, base_entry, oid_len, offset_of_oid, bases)?
1848 }
1849 None => bases.content(oid)?.ok_or_else(|| ResolveError::missing_base(oid))?,
1850 };
1851 let mut delta = Vec::with_capacity(e.uncompressed_size as usize);
1852 inflate_into(pack, e, oid_len, &mut delta)?;
1853 let mut out = Vec::new();
1854 apply_delta_into(&base, &delta, &mut out)?;
1855 Ok((kind, out))
1856 }
1857 }
1858}
1859
1860
1861// ════════════════════════════════════════════════════════════════════════════
1862// The LAZY read path: one object, one chain, straight out of the archive
1863// ════════════════════════════════════════════════════════════════════════════
1864
1865/// Bytes of the verbatim archive, addressed **absolutely**.
1866///
1867/// The mmap ([`crate::archive_map`]) answers without a syscall and without a
1868/// copy; a `pread` answers the tail a snapshot does not cover. Both are the
1869/// same bytes, which is why the chain walker below can take either and never
1870/// know which it got.
1871///
1872/// It exists so [`materialize_at`] can live here — beside the delta code it
1873/// reuses — without this module learning what a store is.
1874pub trait ArchiveBytes {
1875 /// Up to `len` bytes at `at`. **Short is not an error**: a probe near the
1876 /// end of the archive legitimately gets fewer, and the parsers below read a
1877 /// grammar rather than a fixed width. Empty means "there is nothing there",
1878 /// which every caller treats as a corrupt back-reference.
1879 fn at(&self, at: u64, len: u64) -> Result<std::borrow::Cow<'_, [u8]>>;
1880}
1881
1882/// A whole `&[u8]` as an archive — what a test and a single-pack caller use.
1883pub struct SliceArchive<'a>(pub &'a [u8]);
1884
1885impl ArchiveBytes for SliceArchive<'_> {
1886 fn at(&self, at: u64, len: u64) -> Result<std::borrow::Cow<'_, [u8]>> {
1887 let start = usize::try_from(at).unwrap_or(usize::MAX).min(self.0.len());
1888 let end = start.saturating_add(usize::try_from(len).unwrap_or(usize::MAX)).min(self.0.len());
1889 Ok(std::borrow::Cow::Borrowed(&self.0[start..end]))
1890 }
1891}
1892
1893/// **What one read actually cost**, measured by the read rather than predicted.
1894///
1895/// The whole point of carrying it: the cache admission decision needs to know
1896/// whether an object was expensive to produce, and the reader that just
1897/// produced it is the only thing in the system that knows for certain. The
1898/// index could carry a depth column instead — git's `.idx` does not, but the
1899/// walk computes one — and it would still be a prediction of what a read will
1900/// cost, while this is the bill.
1901#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
1902pub struct ReadCost {
1903 /// Delta links followed. `0` is a whole object: one inflate of one zlib
1904 /// stream at one offset, which is what makes a direct read cheap enough
1905 /// that caching it would be a waste of somebody's quota.
1906 pub depth: u32,
1907 /// Bytes the chain produced.
1908 pub inflated: u64,
1909 /// The chain bottomed out on an object this archive does not hold, so a
1910 /// [`BaseSource`] paid for it. Rare, and never cheap.
1911 pub external_base: bool,
1912}
1913
1914/// git's own `pack.depth` ceiling doubled. **The depth at which a chain walk
1915/// stops assuming it will terminate and starts proving it** — see
1916/// [`CycleGuard`]. It is not a refusal any more, and the 🔴 below says why.
1917///
1918/// # 🔴 It WAS a refusal, and that refusal failed ordinary pushes
1919///
1920/// MEASURED, oden, 2026-08-30 (lane AI, reproducing lane AD's rig red): 140
1921/// sequential single-commit pushes of one file into one repository. Push #120
1922/// was refused with *"this pack is thin: it deltas against 509be6a7… which is
1923/// not in it, and no base source could produce that object's content … so this
1924/// repository genuinely does not have that object"* — and every one of the 20
1925/// pushes after it was refused too, forever. The object was **in the store, one
1926/// index lookup away, in a pack that had already been absorbed**. The sentence
1927/// was a lie, and the same red reproduces in this crate alone at push 100 of
1928/// 150 (`crate::thin_storm`).
1929///
1930/// The cause is what a verbatim archive does that stock git does not.
1931/// `git index-pack --fix-thin` *completes* a pushed thin pack by appending the
1932/// resolved base into it, so a stored pack's delta chains never leave it and
1933/// stay under the client's own `pack.depth`. §14 stores the client's bytes
1934/// **verbatim** — that is what makes a clone a byte-range copy — so a
1935/// `REF_DELTA` keeps naming a base in an *earlier pack*, and a repository
1936/// pushed one commit at a time grows a chain **one link per push, without
1937/// bound**. At the 101st link this ceiling turned a legitimate chain into
1938/// "corrupt pack", and [`crate::git_ops`]'s base lookup then reported the
1939/// object as ABSENT rather than as unreadable, which is how a store defect
1940/// wore a client's clothes.
1941///
1942/// A count cannot tell a deep chain from a loop. Revisiting an offset can, and
1943/// that is what [`CycleGuard`] does — so a cycle is caught *sooner* (the first
1944/// repeat, wherever it sits) and a legitimate chain is never refused. The
1945/// constant keeps its name and its value; only its job changed.
1946pub const MAX_DELTA_LINKS: usize = 100;
1947
1948/// **A termination proof for a chain walk, paid for only by the chains that
1949/// need one.**
1950///
1951/// Under [`MAX_DELTA_LINKS`] links it is two integer operations and no
1952/// allocation — the shape every ordinary read has always had. Past that depth
1953/// it starts recording the archive offsets it has stood on, and a repeat is a
1954/// cycle *by definition* rather than by a guess about how deep is too deep.
1955/// The set is only ever entered on chains a corrupt pack or a long push history
1956/// produced, so the hot path is unchanged; a legitimate chain of `d` links
1957/// costs `d - MAX_DELTA_LINKS` `u64` slots and nothing else.
1958///
1959/// It is one guard for both walkers (LAW 5): [`materialize_at`] here and
1960/// `GitStore::resolved_type_at` in [`crate::serve`], which followed the same
1961/// links under a hand-written copy of the same ceiling.
1962pub struct CycleGuard {
1963 links: usize,
1964 seen: std::collections::HashSet<u64>,
1965}
1966
1967impl Default for CycleGuard {
1968 fn default() -> Self {
1969 Self::new()
1970 }
1971}
1972
1973impl CycleGuard {
1974 pub fn new() -> Self {
1975 Self {
1976 links: 0,
1977 // Empty, so a chain shallower than the threshold allocates nothing.
1978 seen: std::collections::HashSet::new(),
1979 }
1980 }
1981
1982 /// One link, standing at archive offset `at`. `Err` **only** when this walk
1983 /// has already stood there, which no acyclic chain can do.
1984 pub fn step(&mut self, at: u64) -> Result<()> {
1985 self.links += 1;
1986 if self.links <= MAX_DELTA_LINKS {
1987 return Ok(());
1988 }
1989 if !self.seen.insert(at) {
1990 bail!(
1991 "the delta chain returns to archive offset {at} after {} links — the pack's \
1992 back-references form a cycle, so the entry cannot be reconstructed. This is a \
1993 corrupt pack and not a deep one: depth alone is never refused here, because a \
1994 verbatim archive grows one link per push (see MAX_DELTA_LINKS)",
1995 self.links
1996 );
1997 }
1998 Ok(())
1999 }
2000
2001 /// How many links this walk has taken.
2002 pub fn links(&self) -> usize {
2003 self.links
2004 }
2005}
2006
2007/// Enough of an entry to hold its type/size varint plus either an `OFS_DELTA`
2008/// distance or a `REF_DELTA` oid. 64 covers a sha256 oid and the widest varint.
2009const HEADER_PROBE: u64 = 64;
2010
2011/// How much compressed stream is fetched per inflate step. The stream's length
2012/// is not written down anywhere the index keeps, so it is discovered by
2013/// inflating; a window means a 40-byte commit costs one probe rather than a
2014/// megabyte of page faults.
2015const INFLATE_WINDOW: u64 = 256 << 10;
2016
2017/// One link of a delta chain, as the entry's own grammar states it.
2018struct Link {
2019 /// The entry's own archive offset — the key a [`ChainCache`] stores it under.
2020 entry_at: u64,
2021 obj_type: ObjType,
2022 /// The size the header declares: of the delta instruction stream for a
2023 /// delta, of the object itself for a whole entry.
2024 size: u64,
2025 /// Where the zlib stream starts — header skipped.
2026 data_at: u64,
2027}
2028
2029/// **One object's content, with only its own chain inflated, addressed by
2030/// archive offset.**
2031///
2032/// This is the lazy-explode read path (`ideas/lazy-explode-index-only-at-push.md`).
2033/// The eager tier's alternative was to read the *whole containing pack* back and
2034/// re-resolve it — for `torvalds/linux` an 8.12 GiB read and a 13.9 M-object
2035/// resolve **per object served**, which is why the exploded tier had to be built
2036/// eagerly at push time and why that tier reached 141.7 GiB. Reading a chain
2037/// instead costs `d` inflates and `d − 1` delta applications and touches nothing
2038/// else in the pack.
2039///
2040/// * `at` — the entry's absolute archive offset, from the `objects` table's
2041/// `offset` column (or from a pending pack's scan).
2042/// * `offset_of_oid` — resolves a `REF_DELTA`'s base oid to an archive offset;
2043/// the `objects` index answers it.
2044/// * `bases` — answers a `REF_DELTA` base the index does not hold (a thin
2045/// pack's external base). `None` from it is
2046/// [`ResolveError::missing_base`], refused by name, never guessed.
2047///
2048/// Nothing is cached and nothing is written: the caller decides whether the
2049/// bytes are worth keeping. That separation is what lets the write-back happen
2050/// off the reader's path.
2051pub fn materialize_at(
2052 archive: &dyn ArchiveBytes,
2053 at: u64,
2054 oid_len: usize,
2055 offset_of_oid: &dyn Fn(&[u8]) -> Option<u64>,
2056 bases: &dyn BaseSource,
2057) -> Result<(GitObjectKind, Vec<u8>, ReadCost)> {
2058 materialize_at_with(archive, at, oid_len, offset_of_oid, bases, None)
2059}
2060
2061/// **A bounded memo of materialised entries by archive offset**, for a reader
2062/// that walks many neighbouring chains — the reachability builder reads
2063/// (nearly) every tree of a repository once, and in a packed repository most
2064/// trees are deltas against the tree before them. Without it a chain of depth
2065/// d costs d inflates and d−1 delta applications per object, every time
2066/// (MEASURED, git.git on dwarf: 330 535 tree reads, 27 s of a 33 s build).
2067/// With it the outward walk stops at the first link already materialised.
2068///
2069/// FIFO by bytes, not LRU: the bases a packed chain reaches are the entries
2070/// written just before it, so recency of insertion is the right order. An
2071/// archive is append-only, so an offset's bytes never change under the memo;
2072/// a gc rewrites into a NEW generation, and this is never held across one.
2073pub struct ChainCache {
2074 map: std::collections::HashMap<u64, (GitObjectKind, Vec<u8>)>,
2075 order: std::collections::VecDeque<u64>,
2076 bytes: usize,
2077 cap: usize,
2078 hits: u64,
2079 misses: u64,
2080}
2081
2082impl ChainCache {
2083 pub fn new(cap_bytes: usize) -> Self {
2084 Self {
2085 map: std::collections::HashMap::new(),
2086 order: std::collections::VecDeque::new(),
2087 bytes: 0,
2088 cap: cap_bytes,
2089 hits: 0,
2090 misses: 0,
2091 }
2092 }
2093
2094 fn get(&mut self, at: u64) -> Option<(GitObjectKind, &[u8])> {
2095 match self.map.get(&at) {
2096 Some((k, p)) => {
2097 self.hits += 1;
2098 Some((*k, p.as_slice()))
2099 }
2100 None => {
2101 self.misses += 1;
2102 None
2103 }
2104 }
2105 }
2106
2107 fn put(&mut self, at: u64, kind: GitObjectKind, payload: &[u8]) {
2108 if payload.len() > self.cap || self.map.contains_key(&at) {
2109 return;
2110 }
2111 while self.bytes + payload.len() > self.cap {
2112 let Some(old) = self.order.pop_front() else { break };
2113 if let Some((_, p)) = self.map.remove(&old) {
2114 self.bytes -= p.len();
2115 }
2116 }
2117 self.bytes += payload.len();
2118 self.order.push_back(at);
2119 self.map.insert(at, (kind, payload.to_vec()));
2120 }
2121
2122 /// `(hits, misses)` since the memo was made.
2123 pub fn stats(&self) -> (u64, u64) {
2124 (self.hits, self.misses)
2125 }
2126
2127 pub fn bytes(&self) -> usize {
2128 self.bytes
2129 }
2130}
2131
2132/// [`materialize_at`] with an optional [`ChainCache`]: the walk outward stops
2133/// at the first link the memo holds, and every link applied on the way back
2134/// down is put in it.
2135pub fn materialize_at_with(
2136 archive: &dyn ArchiveBytes,
2137 at: u64,
2138 oid_len: usize,
2139 offset_of_oid: &dyn Fn(&[u8]) -> Option<u64>,
2140 bases: &dyn BaseSource,
2141 mut cache: Option<&mut ChainCache>,
2142) -> Result<(GitObjectKind, Vec<u8>, ReadCost)> {
2143 // ── walk the chain outward, deepest link last ──────────────────────────
2144 let mut chain: Vec<Link> = Vec::new();
2145 // Set when the chain bottoms out at an object this archive does not hold.
2146 let mut external: Option<Vec<u8>> = None;
2147 // Set when the chain bottoms out at an entry the memo already holds.
2148 let mut memo_hit: Option<(GitObjectKind, Vec<u8>)> = None;
2149 let mut cursor = at;
2150 let mut guard = CycleGuard::new();
2151 loop {
2152 guard.step(cursor)?;
2153 if let Some(c) = cache.as_deref_mut() {
2154 if let Some((k, p)) = c.get(cursor) {
2155 memo_hit = Some((k, p.to_vec()));
2156 break;
2157 }
2158 }
2159 let head = archive.at(cursor, HEADER_PROBE)?;
2160 if head.is_empty() {
2161 bail!("the archive holds no entry at offset {cursor}");
2162 }
2163 let (t, size, n) = crate::pack_walk::type_and_size_of(&head)?;
2164 match t {
2165 ObjType::Commit | ObjType::Tree | ObjType::Blob | ObjType::Tag => {
2166 chain.push(Link { entry_at: cursor, obj_type: t, size, data_at: cursor + n as u64 });
2167 break;
2168 }
2169 ObjType::OfsDelta => {
2170 let (distance, d) = crate::pack_walk::ofs_distance_of(head.get(n..).unwrap_or(&[]))?;
2171 let base = cursor.checked_sub(distance).ok_or_else(|| {
2172 anyhow!(
2173 "the ofs-delta at archive offset {cursor} names a base {distance} bytes \
2174 back, which is before the start of the archive"
2175 )
2176 })?;
2177 chain.push(Link { entry_at: cursor, obj_type: t, size, data_at: cursor + (n + d) as u64 });
2178 cursor = base;
2179 }
2180 ObjType::RefDelta => {
2181 let base_oid = head
2182 .get(n..n + oid_len)
2183 .ok_or_else(|| {
2184 anyhow!(
2185 "the ref-delta entry at archive offset {cursor} has no base oid after \
2186 its header"
2187 )
2188 })?
2189 .to_vec();
2190 chain.push(Link { entry_at: cursor, obj_type: t, size, data_at: cursor + (n + oid_len) as u64 });
2191 match offset_of_oid(&base_oid) {
2192 Some(o) => cursor = o,
2193 None => {
2194 external = Some(base_oid);
2195 break;
2196 }
2197 }
2198 }
2199 }
2200 }
2201 // ── the bottom of the chain: a whole object, or somebody else's ────────
2202 let external_base = external.is_some();
2203 let (kind, mut base) = match (memo_hit, external) {
2204 (Some(hit), _) => hit,
2205 (None, Some(oid)) => bases
2206 .content(&oid)?
2207 .ok_or_else(|| ResolveError::missing_base(&oid))?,
2208 (None, None) => {
2209 let last = chain.pop().expect("a chain that ended has a last link");
2210 let mut out = Vec::new();
2211 inflate_from(archive, last.data_at, last.size, &mut out)?;
2212 let kind = whole_kind(last.obj_type)?;
2213 if let Some(c) = cache.as_deref_mut() {
2214 c.put(last.entry_at, kind, &out);
2215 }
2216 (kind, out)
2217 }
2218 };
2219
2220 // ── apply the deltas back down to the object that was asked for ────────
2221 let depth = chain.len() as u32;
2222 let mut delta = Vec::new();
2223 let mut out = Vec::new();
2224 while let Some(link) = chain.pop() {
2225 inflate_from(archive, link.data_at, link.size, &mut delta)?;
2226 out.clear();
2227 apply_delta_into(&base, &delta, &mut out)?;
2228 std::mem::swap(&mut base, &mut out);
2229 if let Some(c) = cache.as_deref_mut() {
2230 c.put(link.entry_at, kind, &base);
2231 }
2232 }
2233 let cost = ReadCost {
2234 depth,
2235 inflated: base.len() as u64,
2236 external_base,
2237 };
2238 Ok((kind, base, cost))
2239}
2240
2241/// Inflate `expected` bytes of zlib starting at `data_at`, a window at a time.
2242///
2243/// The compressed length is **not** knowable from the index — `objects.len` is
2244/// the entry's stored extent only for an entry the walk measured, and a chain
2245/// base reached by back-reference was not measured here. So the stream is its
2246/// own terminator: inflate until it says `StreamEnd` or until exactly the
2247/// declared number of bytes have come out, and disagree loudly if the two
2248/// answers differ. `out` is reserved once at the declared size and never grows.
2249fn inflate_from(
2250 archive: &dyn ArchiveBytes,
2251 data_at: u64,
2252 expected: u64,
2253 out: &mut Vec<u8>,
2254) -> Result<()> {
2255 let want = usize::try_from(expected).map_err(|_| anyhow!("object size overflows usize"))?;
2256 out.clear();
2257 out.reserve(want);
2258 let mut d = flate2::Decompress::new(true);
2259 let mut pos = data_at;
2260 while out.len() < want {
2261 let chunk = archive.at(pos, INFLATE_WINDOW)?;
2262 if chunk.is_empty() {
2263 bail!("the zlib stream at {data_at} runs past the end of the archive");
2264 }
2265 let before = d.total_in();
2266 let status = d
2267 .decompress_vec(&chunk, out, flate2::FlushDecompress::None)
2268 .map_err(|e| anyhow!("inflating the entry at {data_at}: {e}"))?;
2269 let consumed = d.total_in() - before;
2270 pos += consumed;
2271 if status == flate2::Status::StreamEnd {
2272 break;
2273 }
2274 if consumed == 0 {
2275 bail!(
2276 "the zlib stream at {data_at} stalled after {} of {expected} bytes",
2277 out.len()
2278 );
2279 }
2280 }
2281 if out.len() as u64 != expected {
2282 bail!(
2283 "the entry at {data_at} inflated to {} bytes, its header declared {expected}",
2284 out.len()
2285 );
2286 }
2287 Ok(())
2288}
2289
2290/// The real git type of a non-delta entry.
2291fn whole_kind(t: ObjType) -> Result<GitObjectKind> {
2292 Ok(match t {
2293 ObjType::Commit => GitObjectKind::Commit,
2294 ObjType::Tree => GitObjectKind::Tree,
2295 ObjType::Blob => GitObjectKind::Blob,
2296 ObjType::Tag => GitObjectKind::Tag,
2297 ObjType::OfsDelta | ObjType::RefDelta => {
2298 bail!("a delta entry has no type of its own — its base's type is the answer")
2299 }
2300 })
2301}
2302
2303/// Inflate one entry's stream into `out` (cleared first). The walk already
2304/// proved the length, so this reserves exactly the declared size and never
2305/// grows; a reused `out` allocates nothing.
2306fn inflate_into(pack: &[u8], e: &PackEntry, oid_len: usize, out: &mut Vec<u8>) -> Result<()> {
2307 let start = e.offset as usize;
2308 let end = start + e.len as usize;
2309 if end > pack.len() {
2310 bail!("entry at {start} runs past the end of the pack");
2311 }
2312 // Skip the header the walk already parsed: the zlib stream is what is left
2313 // after the type/size varint and any base reference.
2314 let header = header_len(&pack[start..end], e, oid_len)?;
2315 out.clear();
2316 out.reserve(e.uncompressed_size as usize);
2317 let mut d = flate2::Decompress::new(true);
2318 d.decompress_vec(&pack[start + header..end], out, flate2::FlushDecompress::Finish)
2319 .map_err(|err| anyhow!("inflating the entry at {start}: {err}"))?;
2320 if out.len() as u64 != e.uncompressed_size {
2321 bail!(
2322 "the entry at {start} inflated to {} bytes, the walk measured {}",
2323 out.len(),
2324 e.uncompressed_size
2325 );
2326 }
2327 Ok(())
2328}
2329
2330/// Inflate one entry's stream into a fresh buffer. What a test or a one-off
2331/// caller wants; the resolver uses [`inflate_into`].
2332#[cfg(test)]
2333fn inflate(pack: &[u8], e: &PackEntry, oid_len: usize) -> Result<Vec<u8>> {
2334 let mut out = Vec::new();
2335 inflate_into(pack, e, oid_len, &mut out)?;
2336 Ok(out)
2337}
2338
2339/// How many bytes of an entry are header rather than zlib stream. Recomputed
2340/// from the bytes rather than remembered, so it cannot drift from the walk's own
2341/// reading of them — and the walk's `len` is what bounds it.
2342fn header_len(entry: &[u8], e: &PackEntry, oid_len: usize) -> Result<usize> {
2343 let mut i = 0usize;
2344 let mut cont = true;
2345 while cont {
2346 let b = *entry
2347 .get(i)
2348 .ok_or_else(|| anyhow!("the type/size header runs off the entry"))?;
2349 cont = b & 0x80 != 0;
2350 i += 1;
2351 }
2352 match e.obj_type {
2353 ObjType::OfsDelta => {
2354 let mut cont = true;
2355 while cont {
2356 let b = *entry
2357 .get(i)
2358 .ok_or_else(|| anyhow!("the ofs-delta distance runs off the entry"))?;
2359 cont = b & 0x80 != 0;
2360 i += 1;
2361 }
2362 }
2363 ObjType::RefDelta => i += oid_len,
2364 _ => {}
2365 }
2366 if i >= entry.len() {
2367 bail!("the entry's header consumes all of it, leaving no stream");
2368 }
2369 Ok(i)
2370}
2371
2372/// git's delta format: two sizes, then copy-from-base and insert-literal
2373/// instructions. Fresh-buffer form of [`apply_delta_into`].
2374#[cfg(test)]
2375fn apply_delta(base: &[u8], delta: &[u8]) -> Result<Vec<u8>> {
2376 let mut out = Vec::new();
2377 apply_delta_into(base, delta, &mut out)?;
2378 Ok(out)
2379}
2380
2381/// git's delta format applied into `out` (cleared first, reserved to the
2382/// declared target size).
2383fn apply_delta_into(base: &[u8], delta: &[u8], out: &mut Vec<u8>) -> Result<()> {
2384 let mut i = 0usize;
2385 let base_size = delta_varint(delta, &mut i)?;
2386 if base_size != base.len() as u64 {
2387 bail!(
2388 "the delta expects a base of {base_size} bytes, its base is {}",
2389 base.len()
2390 );
2391 }
2392 let target_size = delta_varint(delta, &mut i)?;
2393 out.clear();
2394 out.reserve(target_size as usize);
2395
2396 while i < delta.len() {
2397 let op = delta[i];
2398 i += 1;
2399 if op & 0x80 != 0 {
2400 // Copy from the base. Offset in up to 4 bytes, size in up to 3;
2401 // a zero size means 0x10000, which is git's own special case.
2402 let mut off = 0u64;
2403 for bit in 0..4 {
2404 if op & (1 << bit) != 0 {
2405 off |= u64::from(*delta.get(i).ok_or_else(|| anyhow!("delta ends mid-copy"))?)
2406 << (bit * 8);
2407 i += 1;
2408 }
2409 }
2410 let mut size = 0u64;
2411 for bit in 0..3 {
2412 if op & (0x10 << bit) != 0 {
2413 size |= u64::from(*delta.get(i).ok_or_else(|| anyhow!("delta ends mid-copy"))?)
2414 << (bit * 8);
2415 i += 1;
2416 }
2417 }
2418 if size == 0 {
2419 size = 0x1_0000;
2420 }
2421 let from = off as usize;
2422 let to = from
2423 .checked_add(size as usize)
2424 .ok_or_else(|| anyhow!("a delta copy range overflows"))?;
2425 if to > base.len() {
2426 bail!(
2427 "a delta copies base[{from}..{to}] out of a {}-byte base",
2428 base.len()
2429 );
2430 }
2431 out.extend_from_slice(&base[from..to]);
2432 } else {
2433 // Insert literal. A zero-length insert is not a valid instruction.
2434 let n = op as usize;
2435 if n == 0 {
2436 bail!("a delta carries a zero-length insert instruction");
2437 }
2438 let end = i
2439 .checked_add(n)
2440 .ok_or_else(|| anyhow!("a delta insert overflows"))?;
2441 if end > delta.len() {
2442 bail!("a delta insert of {n} bytes runs off the end");
2443 }
2444 out.extend_from_slice(&delta[i..end]);
2445 i = end;
2446 }
2447 }
2448
2449 if out.len() as u64 != target_size {
2450 bail!(
2451 "the delta declares a {target_size}-byte result and produced {}",
2452 out.len()
2453 );
2454 }
2455 Ok(())
2456}
2457
2458/// The delta header's little-endian 7-bit varint. Not the pack entry's varint.
2459fn delta_varint(b: &[u8], i: &mut usize) -> Result<u64> {
2460 let mut v = 0u64;
2461 let mut shift = 0u32;
2462 loop {
2463 let byte = *b
2464 .get(*i)
2465 .ok_or_else(|| anyhow!("a delta size varint runs off the end"))?;
2466 *i += 1;
2467 if shift >= 64 {
2468 bail!("a delta size varint is longer than a u64");
2469 }
2470 v |= u64::from(byte & 0x7f) << shift;
2471 shift += 7;
2472 if byte & 0x80 == 0 {
2473 return Ok(v);
2474 }
2475 }
2476}
2477
2478#[cfg(test)]
2479mod tests {
2480 use super::*;
2481 use crate::object::canonical;
2482 use std::path::{Path, PathBuf};
2483
2484 /// A label that answers `.display()`, so a source that is a generated pack
2485 /// and one that is a path on disk name themselves the same way in a failure.
2486 struct Named<'a>(&'a str);
2487 impl Named<'_> {
2488 fn display(&self) -> &str {
2489 self.0
2490 }
2491 }
2492
2493 /// Parse a real `.idx` (v2) into `(oid, pack offset)` pairs. Twenty lines of
2494 /// a frozen format, and it is git's own answer to the question this module
2495 /// computes — which is what makes it a differential guard and not a
2496 /// restatement of our own code.
2497 fn read_idx(bytes: &[u8]) -> Vec<(Vec<u8>, u64)> {
2498 assert_eq!(&bytes[0..4], b"\xfftOc", "not an idx v2");
2499 assert_eq!(u32::from_be_bytes(bytes[4..8].try_into().unwrap()), 2);
2500 let fanout_end = 8 + 256 * 4;
2501 let n = u32::from_be_bytes(bytes[fanout_end - 4..fanout_end].try_into().unwrap()) as usize;
2502 let oids = fanout_end;
2503 let crcs = oids + n * 20;
2504 let offs = crcs + n * 4;
2505 let big = offs + n * 4;
2506 let mut out = Vec::with_capacity(n);
2507 for i in 0..n {
2508 let oid = bytes[oids + i * 20..oids + i * 20 + 20].to_vec();
2509 let raw = u32::from_be_bytes(bytes[offs + i * 4..offs + i * 4 + 4].try_into().unwrap());
2510 let offset = if raw & 0x8000_0000 != 0 {
2511 let j = (raw & 0x7fff_ffff) as usize;
2512 u64::from_be_bytes(bytes[big + j * 8..big + j * 8 + 8].try_into().unwrap())
2513 } else {
2514 u64::from(raw)
2515 };
2516 out.push((oid, offset));
2517 }
2518 out
2519 }
2520
2521 /// Real packs with their real indexes, from this machine's repositories.
2522 fn real_pairs(cap: usize) -> Vec<(PathBuf, PathBuf)> {
2523 let mut out = Vec::new();
2524 let Ok(repos) = std::fs::read_dir(std::path::PathBuf::from(std::env::var_os("HOME").unwrap_or_default()).join("git")) else {
2525 return out;
2526 };
2527 for repo in repos.flatten() {
2528 let dir = repo.path().join(".git/objects/pack");
2529 let Ok(files) = std::fs::read_dir(&dir) else {
2530 continue;
2531 };
2532 for f in files.flatten() {
2533 let p = f.path();
2534 if p.extension().is_some_and(|e| e == "pack")
2535 && f.metadata().map(|m| m.len() < 32 << 20).unwrap_or(false)
2536 {
2537 let idx = p.with_extension("idx");
2538 if idx.exists() {
2539 out.push((p, idx));
2540 if out.len() >= cap {
2541 return out;
2542 }
2543 }
2544 }
2545 }
2546 }
2547 out
2548 }
2549
2550 /// **Pack/index pairs to compare against, the generated corpus first.**
2551 ///
2552 /// 🔴 This used to be [`real_pairs`] alone, and that made the guard's corpus
2553 /// the developer's disk: it took the first six `.pack` files with an `.idx`
2554 /// beside them under `~/git`, in `read_dir` order, and asserted
2555 /// it had found at least one. On a fresh clone, on CI, or on anyone else's
2556 /// machine it fails with *"no real pack found under ~/git — this
2557 /// guard has nothing to compare"* — proven 2026-08-25 by running the suite
2558 /// with a tmpfs over that directory.
2559 ///
2560 /// The corpus from [`crate::pack_fixture`] leads the list, and stock git
2561 /// writes its `.idx` — so the arbiter is still git and not us, but the pack
2562 /// it arbitrates is one this repository states in source. Real packs still
2563 /// follow when the machine has any: they are the only place a **thin** pack
2564 /// turns up, and finding those in the wild is a fact this guard records.
2565 fn sources() -> Vec<(String, Vec<u8>, Vec<u8>)> {
2566 let mut out = Vec::new();
2567 if std::process::Command::new("git")
2568 .arg("--version")
2569 .output()
2570 .is_ok()
2571 {
2572 let pack = crate::pack_fixture::corpus().pack.clone();
2573 let scratch = crate::store::tests::tmpdir("resolve-idx-oracle");
2574 let idx = crate::git_oracle::git_idx_for(&scratch, "corpus.git", &pack)
2575 .expect("stock git indexes the generated corpus");
2576 out.push(("the generated corpus".to_string(), pack, idx));
2577 }
2578 for (pack_path, idx_path) in real_pairs(6) {
2579 let (Ok(pack), Ok(idx)) = (std::fs::read(&pack_path), std::fs::read(&idx_path)) else {
2580 continue;
2581 };
2582 out.push((pack_path.display().to_string(), pack, idx));
2583 }
2584 out
2585 }
2586
2587 /// **git's own `.idx` is the arbiter.** For every real pack on this machine,
2588 /// every oid we compute and the offset we computed it at must appear in
2589 /// git's index, and the two sets must have the same size. A delta applier
2590 /// that is subtly wrong produces a different hash and cannot pass this.
2591 ///
2592 /// Seen RED by changing `size = 0x1_0000` to `size = 0x1_000` in
2593 /// [`apply_delta`] — git's zero-size copy special case:
2594 /// ".../facett/...pack-62582dd0…: the delta declares a 149392-byte result and
2595 /// produced 87952". A single wrong constant in the delta applier and a real
2596 /// repository's pack stops resolving.
2597 ///
2598 /// MEASURED while writing it: **10 899 objects over 3 packs** match git's own
2599 /// `.idx` oid for oid and offset for offset, and **3 of the 6 real packs on
2600 /// this machine are thin** — they delta against objects that are not in them,
2601 /// which is the §14 hole showing up in the wild rather than in theory.
2602 #[test]
2603 fn every_oid_we_compute_is_the_oid_git_wrote_in_its_idx() {
2604 let pairs = sources();
2605 assert!(
2606 !pairs.is_empty(),
2607 "neither stock git nor a real pack is available, so this guard has nothing to \
2608 compare and must not pass silently"
2609 );
2610 let mut objects = 0usize;
2611 let mut full = 0usize;
2612 let mut thin = 0usize;
2613 for (pack_path, pack, idx_bytes) in pairs {
2614 let pack_path = Named(&pack_path);
2615 let idx = read_idx(&idx_bytes);
2616 // MEASURED, and a surprise worth recording: real packs on this
2617 // machine DO delta against objects outside themselves. Such a pack
2618 // cannot be compared here — resolving it is exactly what §14 leaves
2619 // open — but it must not make the guard pass vacuously either, so it
2620 // is counted and named and at least one pack has to compare in full.
2621 let ours = match resolve(&pack, GitHashKind::Sha1, 0, &NoBases) {
2622 Ok(r) => r,
2623 Err(e) => {
2624 assert!(
2625 e.to_string().contains("thin"),
2626 "{}: {e}",
2627 pack_path.display()
2628 );
2629 eprintln!("{}: THIN, external base — skipped", pack_path.display());
2630 thin += 1;
2631 continue;
2632 }
2633 };
2634 full += 1;
2635
2636 assert_eq!(
2637 ours.len(),
2638 idx.len(),
2639 "{}: we resolved {} objects, git indexed {}",
2640 pack_path.display(),
2641 ours.len(),
2642 idx.len()
2643 );
2644 let theirs: HashMap<Vec<u8>, u64> = idx.into_iter().collect();
2645 for r in &ours {
2646 match theirs.get(&r.oid) {
2647 Some(&off) => assert_eq!(
2648 off,
2649 r.offset,
2650 "{}: {} is at {} in git's idx and we put it at {}",
2651 pack_path.display(),
2652 hex::encode(&r.oid),
2653 off,
2654 r.offset
2655 ),
2656 None => panic!(
2657 "{}: we computed {} at offset {}, which git's idx does not contain — the \
2658 resolution is wrong",
2659 pack_path.display(),
2660 hex::encode(&r.oid),
2661 r.offset
2662 ),
2663 }
2664 }
2665 eprintln!("{}: {} objects agree with git", pack_path.display(), ours.len());
2666 objects += ours.len();
2667 }
2668 eprintln!("{objects} objects over {full} packs agree with git's own .idx; {thin} thin");
2669 assert!(full > 0, "every pack compared was thin — nothing was compared");
2670 assert!(objects > 1000, "only {objects} objects compared");
2671 }
2672
2673 /// The three facts the objects table needs and `.idx` cannot supply: the
2674 /// **stored** type (delta or not), the **resolved** type, and the
2675 /// post-resolution size. Asserted against the pack's own delta structure, on
2676 /// a real pack, so it cannot be satisfied by a store-and-echo.
2677 ///
2678 /// Seen RED by resolving a delta to `GitObjectKind::Blob` instead of to its
2679 /// base's kind: "a delta resolves to its base's type:
2680 /// c6d50e761d025c179ffef1c3d68c56aae0a53270 vs base
2681 /// 8e15741296ebe3a0508522b76fe2ec4982396b6d — left: Blob, right: Commit".
2682 ///
2683 /// MEASURED on that pack: 4292 deltas, 2715 of them resolving to something
2684 /// other than a blob, and **all 4292** carrying a resolved size unlike their
2685 /// delta-stream size.
2686 #[test]
2687 fn a_delta_carries_its_stored_type_and_its_resolved_type_and_size() {
2688 let pack = crate::pack_fixture::corpus().pack.clone();
2689 let pack_path = Named("the generated corpus");
2690 let ours = resolve(&pack, GitHashKind::Sha1, 0, &NoBases).unwrap();
2691 // One walk, reused. Walking per entry turns this guard into an O(n^2)
2692 // pass over a 7000-object pack and it stops finishing.
2693 let stream_size: HashMap<u64, u64> = walk(&pack, 20)
2694 .unwrap()
2695 .entries
2696 .iter()
2697 .map(|e| (e.offset, e.uncompressed_size))
2698 .collect();
2699
2700 let deltas: Vec<&Resolved> = ours
2701 .iter()
2702 .filter(|r| matches!(r.stored_type, ObjType::OfsDelta | ObjType::RefDelta))
2703 .collect();
2704 assert!(
2705 !deltas.is_empty(),
2706 "{} carries no deltas, so it proves nothing about resolution",
2707 pack_path.display()
2708 );
2709
2710 // A delta's resolved type is a real git type, never a delta code, and it
2711 // matches its base's.
2712 let by_offset: HashMap<u64, &Resolved> = ours.iter().map(|r| (r.offset, r)).collect();
2713 let mut non_blob = 0usize;
2714 for d in &deltas {
2715 assert!(
2716 matches!(
2717 d.kind,
2718 GitObjectKind::Blob
2719 | GitObjectKind::Tree
2720 | GitObjectKind::Commit
2721 | GitObjectKind::Tag
2722 ),
2723 "a resolved type must be a real git type"
2724 );
2725 if d.kind != GitObjectKind::Blob {
2726 non_blob += 1;
2727 }
2728 if d.delta_base != 0 {
2729 let base = by_offset[&d.delta_base];
2730 assert_eq!(
2731 d.kind,
2732 base.kind,
2733 "a delta resolves to its base's type: {} vs base {}",
2734 hex::encode(&d.oid),
2735 hex::encode(&base.oid)
2736 );
2737 // The delta_base column addresses BYTES, and those bytes are an
2738 // entry we also resolved. An ordinal could not be checked like
2739 // this at all.
2740 assert!(
2741 base.offset < d.offset,
2742 "an ofs-delta base is always earlier in the pack"
2743 );
2744 }
2745 }
2746 assert!(
2747 non_blob > 0,
2748 "{} deltas and not one resolved to a tree/commit/tag — the resolved type is not being \
2749 taken from the base",
2750 deltas.len()
2751 );
2752
2753 // The post-resolution size is the payload's real length, not the delta
2754 // stream's. For a delta the two are almost never equal.
2755 let differing = deltas
2756 .iter()
2757 .filter(|d| stream_size[&d.offset] != d.uncompressed_size)
2758 .count();
2759 assert!(
2760 differing > 0,
2761 "not one delta's resolved size differs from its delta-stream size, which cannot be \
2762 true of a real pack — the resolved size is being copied from the header"
2763 );
2764 eprintln!(
2765 "{}: {} deltas, {non_blob} non-blob, {differing} with a resolved size unlike the \
2766 stream size",
2767 pack_path.display(),
2768 deltas.len()
2769 );
2770 }
2771
2772 /// A thin pack is refused **by name**, and the refusal says what decision is
2773 /// missing. It is not resolved to a wrong oid and it is not silently dropped.
2774 ///
2775 /// Seen RED by making the no-progress arm `bail!("stuck")` instead of
2776 /// `ResolveError::missing_base`: "names the base: stuck".
2777 #[test]
2778 fn a_thin_pack_is_refused_by_name_and_says_what_is_missing() {
2779 // One ref-delta against an oid nothing can supply.
2780 let base_oid = vec![0x5a; 20];
2781 let mut pack = b"PACK".to_vec();
2782 pack.extend_from_slice(&2u32.to_be_bytes());
2783 pack.extend_from_slice(&1u32.to_be_bytes());
2784 // A delta whose base is 4 bytes and whose target is 4 bytes: copy all.
2785 let delta = vec![0x04, 0x04, 0x90, 0x04];
2786 pack.push(0x70 | (delta.len() as u8 & 0x0f)); // type 7, size 4
2787 pack.extend_from_slice(&base_oid);
2788 pack.extend_from_slice(&{
2789 use std::io::Write;
2790 let mut e = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::default());
2791 e.write_all(&delta).unwrap();
2792 e.finish().unwrap()
2793 });
2794 pack.extend_from_slice(&[0u8; 20]);
2795
2796 let err = resolve(&pack, GitHashKind::Sha1, 0, &NoBases)
2797 .expect_err("a thin pack with no base source cannot resolve");
2798 let msg = err.to_string();
2799 assert!(msg.contains(&hex::encode(&base_oid)), "names the base: {msg}");
2800 assert!(msg.contains("§14"), "says which decision is missing: {msg}");
2801
2802 // With a source that CAN supply it, the same pack resolves — so the
2803 // refusal is about the missing content, not about ref-deltas.
2804 struct One(Vec<u8>);
2805 impl BaseSource for One {
2806 fn content(&self, oid: &[u8]) -> Result<Option<(GitObjectKind, Vec<u8>)>> {
2807 Ok((oid == self.0.as_slice()).then(|| (GitObjectKind::Blob, b"abcd".to_vec())))
2808 }
2809 }
2810 let ok = resolve(&pack, GitHashKind::Sha1, 0, &One(base_oid)).expect("with a base it works");
2811 assert_eq!(ok.len(), 1);
2812 assert_eq!(ok[0].kind, GitObjectKind::Blob);
2813 assert_eq!(ok[0].uncompressed_size, 4);
2814 assert_eq!(
2815 ok[0].oid,
2816 GitHashKind::Sha1.oid_of(&canonical(GitObjectKind::Blob, b"abcd")),
2817 "the resolved object is the base copied whole, so it hashes as `abcd`"
2818 );
2819 }
2820
2821 /// **The `delta_base` column, materialised — from a real pack, through all
2822 /// three index layouts and the read stack, and back out again.**
2823 ///
2824 /// This is the guard the column was added for. It asserts *applied output*:
2825 /// resolve a real pack, build every arm over the rows, and for every
2826 /// `OFS_DELTA` read the base offset **back out of the index** and require it
2827 /// to land on an object the same index holds — earlier in the archive, and
2828 /// with a byte extent that contains it. An ordinal in that column could not
2829 /// satisfy this at all, which is §13's decision checked rather than quoted.
2830 ///
2831 /// The three arms are compared row for row, because that is what caught a
2832 /// swapped `len`/`size` write in the previous sweep while two of the three
2833 /// stayed green.
2834 ///
2835 /// Seen RED three times, each for a different way the column can be wrong
2836 /// while every other fact stays right:
2837 ///
2838 /// 1. **Never written.** `Resolved::index_entry` returning `delta_base: 0`:
2839 /// "FourTables: b76687fbb34020672608bcb1a5e027c2d8b0346a is an ofs-delta
2840 /// with no recorded base".
2841 /// 2. **Written from the wrong field.** `delta_base: self.offset` — a
2842 /// plausible archive offset that is non-zero, inside the pack, and lands
2843 /// on a real entry boundary, so only the identity check catches it:
2844 /// "FourTables: b76687fb… records base 91272 which is its own offset,
2845 /// not its base's".
2846 /// 3. **Written in the wrong coordinate space.** `resolve_walked` yielding
2847 /// `*o` instead of `o + archive_offset`, i.e. pack-relative rather than
2848 /// archive-relative — the bug that an *ordinal* column would have had no
2849 /// way to express and no way to detect: "FourTables: b76687fb… records
2850 /// base offset 86534, which starts no object in this index". This is why
2851 /// the fixture resolves at `AT = 4096` rather than at 0; at 0 the two
2852 /// spaces coincide and the guard would be blind.
2853 ///
2854 /// All three restored.
2855 #[test]
2856 fn the_delta_base_column_survives_all_three_arms_over_the_corpus() {
2857 use crate::index_layout::{
2858 FourTables, IndexEntry, ObjType, ObjectIndex, OneTableFourColumns, PackedPayload,
2859 };
2860 use crate::read_stack::{ObjectReadStack, RebuildTriggers};
2861
2862 let pack = crate::pack_fixture::corpus().pack.clone();
2863 // A non-zero archive offset on purpose: `delta_base` is in the archive's
2864 // coordinate space, so a resolver that forgot to rebase it would give
2865 // pack-relative values that no longer locate anything.
2866 const AT: u64 = 4096;
2867 let rows = resolve(&pack, GitHashKind::Sha1, AT, &NoBases).unwrap();
2868 let entries: Vec<IndexEntry> = rows.iter().map(|r| r.index_entry()).collect();
2869
2870 let a = FourTables::build(&entries).unwrap();
2871 let b = OneTableFourColumns::build(&entries).unwrap();
2872 let c = PackedPayload::build(&entries).unwrap();
2873 let stack = ObjectReadStack::<OneTableFourColumns>::in_memory(RebuildTriggers::manual())
2874 .unwrap();
2875 stack.append(&entries).unwrap();
2876 stack.rebuild().unwrap();
2877
2878 // Every extent this pack's objects occupy, read back out of the index.
2879 let extents: HashMap<u64, u64> = entries
2880 .iter()
2881 .map(|e| {
2882 let r = a.lookup(&e.oid).expect("a stored oid resolves");
2883 (r.offset, r.len)
2884 })
2885 .collect();
2886 assert_eq!(extents.len(), entries.len(), "two objects share an offset");
2887
2888 let named: [(&str, &dyn ObjectIndex); 4] =
2889 [("FourTables", &a), ("OneTableFourColumns", &b), ("PackedPayload", &c), ("ObjectReadStack", &stack)];
2890 let mut ofs_deltas = 0usize;
2891 for (name, idx) in named {
2892 let mut with_base = 0usize;
2893 for e in &entries {
2894 let row = idx.lookup(&e.oid).expect("a stored oid resolves");
2895 // All four layouts, row for row, on every fact.
2896 assert_eq!(
2897 row,
2898 a.lookup(&e.oid).unwrap(),
2899 "{name} disagrees with FourTables on {}",
2900 hex::encode(&e.oid)
2901 );
2902 if row.obj_type != ObjType::OfsDelta {
2903 continue;
2904 }
2905 with_base += 1;
2906 assert_ne!(
2907 row.delta_base, 0,
2908 "{name}: {} is an ofs-delta with no recorded base",
2909 hex::encode(&e.oid)
2910 );
2911 assert_ne!(
2912 row.delta_base,
2913 row.offset,
2914 "{name}: {} records base {} which is its own offset, not its base's",
2915 hex::encode(&e.oid),
2916 row.delta_base
2917 );
2918 assert!(
2919 row.delta_base >= AT,
2920 "{name}: base {} is below the archive offset {AT} — it was never rebased",
2921 row.delta_base
2922 );
2923 // The whole claim: it addresses an entry this index holds.
2924 let base_len = extents.get(&row.delta_base).unwrap_or_else(|| {
2925 panic!(
2926 "{name}: {} records base offset {}, which starts no object in this index",
2927 hex::encode(&e.oid),
2928 row.delta_base
2929 )
2930 });
2931 assert!(
2932 row.delta_base + base_len <= row.offset,
2933 "{name}: the base at {} (+{base_len}) overlaps the delta at {}",
2934 row.delta_base,
2935 row.offset
2936 );
2937 }
2938 assert!(
2939 with_base > 0,
2940 "{name}: {} ofs-deltas resolved and 0 of them carry a base offset — the column \
2941 is not being written",
2942 entries
2943 .iter()
2944 .filter(|e| e.obj_type == ObjType::OfsDelta)
2945 .count()
2946 );
2947 ofs_deltas = with_base;
2948 }
2949 assert!(
2950 ofs_deltas > 100,
2951 "the corpus carries only {ofs_deltas} ofs-deltas — too few to prove anything"
2952 );
2953 eprintln!(
2954 "{ofs_deltas} ofs-delta base offsets located their base entry in all four layouts"
2955 );
2956 }
2957
2958 /// **A base that several entries delta against survives until the last one
2959 /// has had it.**
2960 ///
2961 /// This replaces `a_blob_that_is_nobodys_base_is_hashed_and_dropped`, which
2962 /// read the retention policy off `Resolved::payload`. That field is gone
2963 /// (see the struct's docs), and the property it stood for — peak memory is
2964 /// the live base set, not the object set — is measured in bytes now by
2965 /// `tests/resolve_peak_memory.rs` rather than inferred from a flag.
2966 ///
2967 /// What is left here is the *risk* the reference counting introduced, and it
2968 /// is the one that matters: an off-by-one that frees a base while a
2969 /// dependent still needs it. That cannot be a silent wrong answer —
2970 /// `resolve_walked` fails the whole pack by name — so the assertion is that
2971 /// a pack full of shared bases resolves at all.
2972 ///
2973 /// **The test refuses to run on a pack that could not expose the bug.** A
2974 /// corpus of only single-dependent chains would pass this with the counting
2975 /// deleted entirely, which is the blind-guard shape LAW 2 is about, so the
2976 /// fan-out is asserted before the resolution is.
2977 ///
2978 /// Seen RED by `*left -= 1;` → `*left = 0;` in [`consume`] (drop every base
2979 /// on first use): "entry at offset 3141 deltas against offset 2724 which no
2980 /// entry starts at — the pack is corrupt", on the first pack tried.
2981 #[test]
2982 fn a_base_many_entries_share_outlives_all_of_them() {
2983 let mut checked = 0usize;
2984 let mut widest = 0usize;
2985 let mut corpora: Vec<(String, Vec<u8>)> =
2986 vec![("the generated corpus".to_string(), crate::pack_fixture::corpus().pack.clone())];
2987 for (pack_path, _) in real_pairs(4) {
2988 if let Ok(bytes) = std::fs::read(&pack_path) {
2989 corpora.push((pack_path.display().to_string(), bytes));
2990 }
2991 }
2992 for (pack_path, pack) in corpora {
2993 let pack_path = Named(&pack_path);
2994 let w = walk(&pack, 20).unwrap();
2995
2996 // How many entries name each base. Anything above 1 is a base the
2997 // counting has to hold across more than one consumer.
2998 let mut uses: HashMap<u64, usize> = HashMap::new();
2999 for e in &w.entries {
3000 if let DeltaBase::Offset(o) = e.delta_base {
3001 *uses.entry(o).or_default() += 1;
3002 }
3003 }
3004 let shared = uses.values().filter(|n| **n > 1).count();
3005 let fan_out = uses.values().copied().max().unwrap_or(0);
3006 if shared == 0 {
3007 continue;
3008 }
3009 widest = widest.max(fan_out);
3010
3011 let rows = resolve(&pack, GitHashKind::Sha1, 0, &NoBases)
3012 .unwrap_or_else(|e| panic!("{}: {e}", pack_path.display()));
3013 assert_eq!(
3014 rows.len(),
3015 w.entries.len(),
3016 "{}: {} of {} entries resolved",
3017 pack_path.display(),
3018 rows.len(),
3019 w.entries.len()
3020 );
3021 eprintln!(
3022 "{}: {shared} bases shared by more than one entry, widest fan-out {fan_out}, all \
3023 {} entries resolved",
3024 pack_path.display(),
3025 rows.len()
3026 );
3027 checked += 1;
3028 }
3029 assert!(
3030 checked > 0 && widest > 2,
3031 "no pack compared had a base shared by more than two entries (checked {checked}, \
3032 widest {widest}) — this test cannot see an early free and must not report a pass"
3033 );
3034 }
3035
3036 /// The corpus plus whatever real packs the box has, as `(name, bytes)`.
3037 fn corpora(cap: usize) -> Vec<(String, Vec<u8>)> {
3038 let mut out: Vec<(String, Vec<u8>)> =
3039 vec![("the generated corpus".to_string(), crate::pack_fixture::corpus().pack.clone())];
3040 for (pack_path, _) in real_pairs(cap) {
3041 if let Ok(bytes) = std::fs::read(&pack_path) {
3042 out.push((pack_path.display().to_string(), bytes));
3043 }
3044 }
3045 out
3046 }
3047
3048 /// Every `(oid, offset, size, delta_base)` the resolver produced, sorted.
3049 fn rows_of(rows: &[Resolved]) -> Vec<(Vec<u8>, u64, u64, u64)> {
3050 let mut v: Vec<_> = rows
3051 .iter()
3052 .map(|r| (r.oid.clone(), r.offset, r.uncompressed_size, r.delta_base))
3053 .collect();
3054 v.sort();
3055 v
3056 }
3057
3058 fn stream(pack: &[u8], limits: ResolveLimits<'_>) -> (Vec<Resolved>, ResolveReport, u64) {
3059 let w = walk(pack, 20).unwrap();
3060 let mut out = Vec::new();
3061 let mut calls = 0u64;
3062 let report = resolve_streaming(
3063 pack,
3064 &w,
3065 GitHashKind::Sha1,
3066 0,
3067 &NoBases,
3068 &NoSink,
3069 limits,
3070 &mut |rows| {
3071 calls += 1;
3072 out.extend(rows.drain(..));
3073 Ok(())
3074 },
3075 )
3076 .unwrap();
3077 (out, report, calls)
3078 }
3079
3080 /// **The bounded base set: same rows, hard ceiling, re-inflations counted.**
3081 ///
3082 /// A budget of one byte is the harshest ceiling the walk can be given: only
3083 /// the base the very next delta reads is held, and every shallower one on
3084 /// the path is dropped and inflated again from the pack. The oids, sizes
3085 /// and delta bases must come out identical to the unbounded pass, the peak
3086 /// must never exceed one base, and the count of re-inflations must be
3087 /// positive — otherwise the ceiling was never hit and the test proved
3088 /// nothing.
3089 ///
3090 /// # What "the ceiling can bite here" means, and why the predicate changed
3091 ///
3092 /// It used to be "some base has more than one dependent", which was the
3093 /// shape the **fixpoint** evicted on: it held every base from its own entry
3094 /// until its last dependent's, so two live bases needed only two bases.
3095 /// The forest walk holds the *path*, so a second live base needs a path two
3096 /// deep with work still to do above it — a node with at least two children
3097 /// one of which is itself a base. A pack without that shape now genuinely
3098 /// never evicts, and asserting that it does would be asserting the old
3099 /// algorithm.
3100 #[test]
3101 fn a_bounded_base_set_yields_the_same_rows_under_a_hard_ceiling_and_counts_reinflations() {
3102 let mut checked = 0;
3103 for (name, pack) in corpora(4) {
3104 let w = walk(&pack, 20).unwrap();
3105 let mut uses: HashMap<u64, usize> = HashMap::new();
3106 for e in &w.entries {
3107 if let DeltaBase::Offset(o) = e.delta_base {
3108 *uses.entry(o).or_default() += 1;
3109 }
3110 }
3111 let forest = crate::forest::DeltaForest::build(&w).unwrap();
3112 let two_deep = (0..w.entries.len() as u32).any(|i| {
3113 let kids = forest.children(i);
3114 kids.len() > 1 && kids[..kids.len() - 1].iter().any(|c| !forest.children(*c).is_empty())
3115 });
3116 if !uses.values().any(|n| *n > 1) || !two_deep {
3117 continue;
3118 }
3119 let (free, free_report, _) = stream(&pack, ResolveLimits::none());
3120 assert_eq!(free_report.reinflations, 0, "{name}: unbounded must never re-inflate");
3121 assert_eq!(free_report.evictions, 0);
3122 let largest_base = free
3123 .iter()
3124 .filter(|r| uses.contains_key(&r.offset))
3125 .map(|r| r.uncompressed_size)
3126 .max()
3127 .unwrap();
3128 let tight = ResolveLimits { base_budget: 1, ..ResolveLimits::none() };
3129 let (bounded, report, _) = stream(&pack, tight);
3130 assert_eq!(rows_of(&free), rows_of(&bounded), "{name}: the ceiling changed the output");
3131 assert!(report.reinflations > 0, "{name}: no re-inflation — the ceiling never bit");
3132 assert!(report.evictions > 0, "{name}: nothing evicted");
3133 assert!(
3134 report.peak_base_bytes <= largest_base,
3135 "{name}: peak held {} > the largest single base {largest_base} — the ceiling leaked",
3136 report.peak_base_bytes
3137 );
3138 assert!(
3139 free_report.peak_base_bytes >= report.peak_base_bytes,
3140 "{name}: the unbounded pass held less ({}) than the bounded one ({})",
3141 free_report.peak_base_bytes,
3142 report.peak_base_bytes
3143 );
3144 // A generous budget: same rows, fewer re-inflations than the
3145 // harsh one — the eviction is Belady, so more room is never worse.
3146 let mid = ResolveLimits { base_budget: largest_base * 3, ..ResolveLimits::none() };
3147 let (mid_rows, mid_report, _) = stream(&pack, mid);
3148 assert_eq!(rows_of(&free), rows_of(&mid_rows), "{name}: mid budget changed the output");
3149 assert!(
3150 mid_report.reinflations <= report.reinflations,
3151 "{name}: more room, more re-inflations ({} > {})",
3152 mid_report.reinflations,
3153 report.reinflations
3154 );
3155 assert!(mid_report.peak_base_bytes <= largest_base * 3);
3156 eprintln!(
3157 "{name}: {} objects, unbounded peak {} B; budget 1 B → peak {} B, {} evictions, {} re-inflations; budget {} B → {} re-inflations",
3158 free.len(),
3159 free_report.peak_base_bytes,
3160 report.peak_base_bytes,
3161 report.evictions,
3162 report.reinflations,
3163 largest_base * 3,
3164 mid_report.reinflations
3165 );
3166 checked += 1;
3167 }
3168 assert!(
3169 checked > 0,
3170 "no pack in the corpus has a path the ceiling can bite on — nothing was proven"
3171 );
3172 }
3173
3174 /// Rows arrive in batches of the size asked for, the last one partial, and
3175 /// nothing is lost or doubled across the batch boundary.
3176 #[test]
3177 fn rows_are_handed_over_in_batches_and_the_union_is_the_whole_pack() {
3178 let pack = crate::pack_fixture::corpus().pack.clone();
3179 let n = walk(&pack, 20).unwrap().entries.len();
3180 assert!(n > 7);
3181 let (all, report, calls) = stream(&pack, ResolveLimits { row_batch: 7, ..ResolveLimits::none() });
3182 assert_eq!(all.len(), n);
3183 assert_eq!(calls, n.div_ceil(7) as u64, "batches of 7 over {n} rows");
3184 assert_eq!(report.batches, calls);
3185 assert_eq!(report.rows, n as u64);
3186 let (one, _, calls1) = stream(&pack, ResolveLimits { row_batch: usize::MAX, ..ResolveLimits::none() });
3187 assert_eq!(calls1, 1);
3188 assert_eq!(rows_of(&all), rows_of(&one));
3189 let mut oids: Vec<_> = all.iter().map(|r| r.oid.clone()).collect();
3190 oids.sort();
3191 oids.dedup();
3192 assert_eq!(oids.len(), n, "a row was doubled across a batch boundary");
3193 }
3194
3195 /// **The watchdog inside a pack**: with a fake meminfo under the floor the
3196 /// resolver drops every held base, flushes, and waits until the fake says
3197 /// memory is back — then finishes with identical rows.
3198 #[test]
3199 fn the_watchdog_pauses_a_resolve_mid_pack_drops_the_bases_and_the_rows_are_unchanged() {
3200 use std::sync::Arc;
3201 use std::time::Duration;
3202 let dir = crate::store::tests::tmpdir("resolve-watchdog");
3203 let meminfo = dir.join("meminfo");
3204 let write = |avail_kib: u64| {
3205 std::fs::write(
3206 &meminfo,
3207 format!("MemTotal: 16000000 kB\nMemAvailable: {avail_kib} kB\n"),
3208 )
3209 .unwrap()
3210 };
3211 write(1_000_000);
3212 let dog = Arc::new(MemWatchdog::with(&meminfo, 3_200_000 * 1024, Duration::from_millis(5)));
3213 let pack = crate::pack_fixture::corpus().pack.clone();
3214 let (free, _, _) = stream(&pack, ResolveLimits::none());
3215 let m2 = meminfo.clone();
3216 // **Raise only once the pause is actually open.** Sleeping a flat 150 ms
3217 // and hoping the resolver got there first is a race the loaded box
3218 // loses: under a full parallel suite the first checkpoint can land at
3219 // 140 ms, and the assertion below then reads a 10 ms pause. Wait for
3220 // `is_paused`, then hold for the interval the assertion is about.
3221 let d2 = dog.clone();
3222 let raiser = std::thread::spawn(move || {
3223 let deadline = std::time::Instant::now() + Duration::from_secs(30);
3224 while !d2.is_paused() && std::time::Instant::now() < deadline {
3225 std::thread::sleep(Duration::from_millis(2));
3226 }
3227 std::thread::sleep(Duration::from_millis(150));
3228 std::fs::write(&m2, "MemTotal: 16000000 kB\nMemAvailable: 9000000 kB\n").unwrap();
3229 });
3230 let limits = ResolveLimits {
3231 check_every: 3,
3232 watchdog: Some(dog.clone()),
3233 ..ResolveLimits::none()
3234 };
3235 let (rows, report, _) = stream(&pack, limits);
3236 raiser.join().unwrap();
3237 assert_eq!(rows_of(&free), rows_of(&rows), "the pause changed the output");
3238 assert_eq!(report.pauses, 1, "one pause: the floor was breached once and the raise ended it");
3239 assert!(report.watchdog_drops >= 1, "the pause must drop the held bases");
3240 assert!(report.paused_ms >= 100, "waited {} ms; must have waited for the raise", report.paused_ms);
3241 assert_eq!(dog.pauses(), 1);
3242 }
3243
3244 /// **A short box loses workers, not the pack.**
3245 ///
3246 /// The watchdog says the floor is breached and never un-says it. With one
3247 /// worker that is a pause and then a refusal by name. With four it must
3248 /// first be a *shed*: every worker above the lowest unit in flight gives
3249 /// its core back and waits, the round drains onto one core, and the pack
3250 /// still comes out — byte for byte the same stream a one-worker run makes.
3251 ///
3252 /// MEASURED on the rig and this is what it is for: the 16 GB appliance
3253 /// finishes the 8.12 GiB kernel mirror at one worker with 5.55 GiB to
3254 /// spare, and refuses it at 96 % with two and 86 % with four.
3255 #[test]
3256 fn a_short_box_sheds_its_workers_before_it_refuses_the_pack() {
3257 use std::sync::Arc;
3258 use std::time::Duration;
3259 let dir = crate::store::tests::tmpdir("resolve-shed");
3260 let meminfo = dir.join("meminfo");
3261 // Breached from the first checkpoint, and raised only once a worker HAS
3262 // been shed and the last one standing has opened a pause — which is
3263 // the shape the rig measured: the box was short because of the extra
3264 // workers, and giving them back is what makes room.
3265 //
3266 // 🔴 It raised on the first pause alone, and that was a race the test
3267 // lost under load (2026-10-16, dwarf at load 20-30: 1 red in 5 runs of
3268 // the module, never alone): a pause opens only when the lowest unit is
3269 // the ONLY one in flight, so if the other three workers had not been
3270 // scheduled onto a unit yet, the box was raised before any of them
3271 // reached a checkpoint, nothing could ever be shed, and the guard said
3272 // "NOTHING was shed". Shedding while the lowest is paused is legitimate
3273 // (it drained before pausing, so the others may enter and then park),
3274 // so the raise now waits for the shed it is there to prove. The
3275 // watchdog's own 30 s bound is what fails a run that never sheds.
3276 std::fs::write(&meminfo, "MemTotal: 16000000 kB\nMemAvailable: 1000000 kB\n").unwrap();
3277 let pack = crate::pack_fixture::corpus().pack.clone();
3278 let one = tape_of(&pack, 1, false).0;
3279 let dog = Arc::new(
3280 MemWatchdog::with(&meminfo, 3_200_000 * 1024, Duration::from_millis(2))
3281 .bounded(1 << 20, Duration::from_secs(30)),
3282 );
3283 let progress = crate::progress::IndexProgress::detached();
3284 let (m2, d2, p2) = (meminfo.clone(), dog.clone(), progress.clone());
3285 let raiser = std::thread::spawn(move || {
3286 let deadline = std::time::Instant::now() + Duration::from_secs(30);
3287 while !(p2.sheds() > 0 && d2.is_paused()) && std::time::Instant::now() < deadline {
3288 std::thread::sleep(Duration::from_millis(2));
3289 }
3290 std::fs::write(&m2, "MemTotal: 16000000 kB\nMemAvailable: 9000000 kB\n").unwrap();
3291 });
3292 let w = walk(&pack, 20).unwrap();
3293 let tape = Tape::default();
3294 let sink = TapeSink { tape: &tape, only_commits: false };
3295 let limits = ResolveLimits {
3296 workers: 4,
3297 row_batch: 64,
3298 lookahead_bytes: 1 << 16,
3299 check_every: 2,
3300 watchdog: Some(dog.clone()),
3301 progress: Some(&progress),
3302 ..ResolveLimits::none()
3303 };
3304 let report = resolve_streaming(
3305 &pack,
3306 &w,
3307 GitHashKind::Sha1,
3308 0,
3309 &NoBases,
3310 &sink,
3311 limits,
3312 &mut |rows| {
3313 tape.put(b'B', &[&(rows.len() as u64).to_le_bytes()]);
3314 for r in rows.iter() {
3315 tape.put(
3316 b'R',
3317 &[
3318 &r.oid,
3319 &r.offset.to_le_bytes(),
3320 &r.len.to_le_bytes(),
3321 &[r.stored_type as u8, r.kind as u8],
3322 &r.uncompressed_size.to_le_bytes(),
3323 &r.delta_base.to_le_bytes(),
3324 ],
3325 );
3326 }
3327 rows.clear();
3328 Ok(())
3329 },
3330 );
3331 raiser.join().unwrap();
3332 let report = report.expect("a shed must not refuse the pack");
3333 assert!(
3334 report.shed > 0,
3335 "the box was short from the first checkpoint and NOTHING was shed — the guard \
3336 proved nothing (workers {}, pauses {})",
3337 report.workers,
3338 report.pauses
3339 );
3340 assert_eq!(report.rows, w.entries.len() as u64, "every entry is still a row");
3341 assert!(tape.take() == one, "the shed changed the stream");
3342 }
3343
3344 /// A cancel flag set mid-pack aborts at the next checkpoint with the named
3345 /// error — the pack is not finished and nothing pretends it was.
3346 #[test]
3347 fn a_cancel_aborts_the_pack_at_the_next_checkpoint() {
3348 let cancel = AtomicBool::new(true);
3349 let pack = crate::pack_fixture::corpus().pack.clone();
3350 let w = walk(&pack, 20).unwrap();
3351 let limits = ResolveLimits { check_every: 2, cancel: Some(&cancel), ..ResolveLimits::none() };
3352 let mut got = 0usize;
3353 let err = resolve_streaming(&pack, &w, GitHashKind::Sha1, 0, &NoBases, &NoSink, limits, &mut |rows| {
3354 got += rows.len();
3355 rows.clear();
3356 Ok(())
3357 })
3358 .unwrap_err();
3359 assert!(err.to_string().contains("cancelled"), "{err:#}");
3360 assert!(got < w.entries.len(), "the cancel did not stop the pack ({got} rows)");
3361 }
3362
3363 // ════════════════════════════════════════════════════════════════════════
3364 // THE PARALLEL RESOLVER — one stream out, whatever the worker count
3365 // ════════════════════════════════════════════════════════════════════════
3366
3367 /// **Everything the resolver hands out, as bytes.**
3368 ///
3369 /// Not a set and not a sorted digest: the *sequence*, with the sink's calls
3370 /// and the row callback's batches interleaved exactly as they happened. A
3371 /// comparison over sorted rows would pass on any ordering at all, and
3372 /// ordering is the whole thing a worker count could change.
3373 #[derive(Default)]
3374 struct Tape {
3375 bytes: std::sync::Mutex<Vec<u8>>,
3376 }
3377
3378 impl Tape {
3379 fn put(&self, tag: u8, parts: &[&[u8]]) {
3380 let mut b = self.bytes.lock().unwrap_or_else(|e| e.into_inner());
3381 b.push(tag);
3382 for p in parts {
3383 b.extend_from_slice(&(p.len() as u64).to_le_bytes());
3384 b.extend_from_slice(p);
3385 }
3386 }
3387 fn take(&self) -> Vec<u8> {
3388 std::mem::take(&mut self.bytes.lock().unwrap_or_else(|e| e.into_inner()))
3389 }
3390 }
3391
3392 /// A sink that records every call — and, on the second arm, one that says it
3393 /// wants only commits, which is the shipped `ExplodePolicy::Commits` shape.
3394 struct TapeSink<'t> {
3395 tape: &'t Tape,
3396 only_commits: bool,
3397 }
3398
3399 impl crate::exploded::PayloadSink for TapeSink<'_> {
3400 fn explode(&self, oid: &[u8], kind: GitObjectKind, payload: &[u8]) -> Result<()> {
3401 self.tape.put(b'E', &[oid, &[kind as u8], payload]);
3402 Ok(())
3403 }
3404 fn wants(&self, kind: GitObjectKind) -> bool {
3405 !self.only_commits || matches!(kind, GitObjectKind::Commit | GitObjectKind::Tag)
3406 }
3407 }
3408
3409 /// One resolve, recorded. `workers` is the only thing that varies.
3410 fn tape_of(pack: &[u8], workers: usize, only_commits: bool) -> (Vec<u8>, ResolveReport) {
3411 let w = walk(pack, 20).unwrap();
3412 let tape = Tape::default();
3413 let sink = TapeSink { tape: &tape, only_commits };
3414 let limits = ResolveLimits {
3415 workers,
3416 // Small enough that a real pack crosses it many times, so the
3417 // batch boundaries are part of what is being compared.
3418 row_batch: 64,
3419 // …and small enough that the lookahead ceiling is actually reached,
3420 // so the backpressure path is on the tape too.
3421 lookahead_bytes: 1 << 16,
3422 ..ResolveLimits::none()
3423 };
3424 let report = resolve_streaming(
3425 pack,
3426 &w,
3427 GitHashKind::Sha1,
3428 0,
3429 &NoBases,
3430 &sink,
3431 limits,
3432 &mut |rows| {
3433 tape.put(b'B', &[&(rows.len() as u64).to_le_bytes()]);
3434 for r in rows.iter() {
3435 tape.put(
3436 b'R',
3437 &[
3438 &r.oid,
3439 &r.offset.to_le_bytes(),
3440 &r.len.to_le_bytes(),
3441 &[r.stored_type as u8, r.kind as u8],
3442 &r.uncompressed_size.to_le_bytes(),
3443 &r.delta_base.to_le_bytes(),
3444 ],
3445 );
3446 }
3447 rows.clear();
3448 Ok(())
3449 },
3450 )
3451 .expect("resolve");
3452 (tape.take(), report)
3453 }
3454
3455 /// **The determinism proof.** The same pack, resolved at 1, 2 and 4
3456 /// workers, produces byte-identical output — the same rows, in the same
3457 /// order, in the same batches, with the same sink calls between them.
3458 #[test]
3459 fn the_same_pack_resolves_to_the_same_bytes_at_1_2_and_4_workers() {
3460 let mut proved = 0usize;
3461 for (name, pack) in corpora(4) {
3462 for only_commits in [false, true] {
3463 let (one, r1) = tape_of(&pack, 1, only_commits);
3464 let (two, r2) = tape_of(&pack, 2, only_commits);
3465 let (four, r4) = tape_of(&pack, 4, only_commits);
3466 assert!(!one.is_empty(), "{name}: the tape is empty, so it proves nothing");
3467 assert_eq!(one.len(), two.len(), "{name}: 1 vs 2 workers, tape length");
3468 assert!(one == two, "{name}: 1 vs 2 workers, byte for byte");
3469 assert!(one == four, "{name}: 1 vs 4 workers, byte for byte");
3470 assert_eq!(r1.rows, r2.rows, "{name}");
3471 assert_eq!(r1.rows, r4.rows, "{name}");
3472 assert_eq!(r1.batches, r4.batches, "{name}");
3473 assert_eq!(r1.workers, 1, "{name}");
3474 proved += 1;
3475 }
3476 }
3477 assert!(proved >= 2, "no pack was compared — this guard proved nothing");
3478 }
3479
3480 /// 🔴 **LAW 2: the comparison above can fail.** If the tape were blind to
3481 /// order — a sorted digest, a set of oids, a row count — every one of those
3482 /// assertions would pass on a resolver that emitted its objects in any
3483 /// order at all. So: take the real 4-worker tape, move ONE row later in it,
3484 /// and prove the comparison rejects it.
3485 #[test]
3486 fn the_determinism_tape_rejects_a_stream_whose_order_moved() {
3487 let pack = corpora(2).into_iter().next().expect("a pack").1;
3488 let (good, _) = tape_of(&pack, 4, false);
3489 // Find the two `R` records the tape starts with and swap them. Nothing
3490 // about the *contents* changes; only the order does.
3491 let a = good.iter().position(|b| *b == b'R').expect("a row record");
3492 let rec = 1 + 6 * 8 + 20 + 8 + 8 + 2 + 8 + 8; // tag + 6 lengths + fields
3493 assert!(good.len() > a + 2 * rec, "the tape is too short to reorder");
3494 let mut bad = good.clone();
3495 for i in 0..rec {
3496 bad.swap(a + i, a + rec + i);
3497 }
3498 assert_ne!(good, bad, "the two rows were identical — swapping them proved nothing");
3499 assert!(good != bad, "an order-blind comparison would have called these equal");
3500 }
3501
3502 /// A sink that answers `wants = false` is **not called** for that kind, and
3503 /// the resolver still produces every row. This is what keeps the ordered
3504 /// hand-over from buffering 90 % of a pack's bytes.
3505 #[test]
3506 fn a_sink_that_wants_nothing_is_never_handed_a_payload() {
3507 for (name, pack) in corpora(3) {
3508 let w = walk(&pack, 20).unwrap();
3509 let tape = Tape::default();
3510 let sink = TapeSink { tape: &tape, only_commits: true };
3511 let mut rows = 0u64;
3512 let report = resolve_streaming(
3513 &pack,
3514 &w,
3515 GitHashKind::Sha1,
3516 0,
3517 &NoBases,
3518 &sink,
3519 ResolveLimits { workers: 4, ..ResolveLimits::none() },
3520 &mut |r| {
3521 rows += r.len() as u64;
3522 r.clear();
3523 Ok(())
3524 },
3525 )
3526 .expect("resolve");
3527 assert_eq!(rows, w.entries.len() as u64, "{name}: every entry is a row");
3528 assert_eq!(report.rows, rows, "{name}");
3529 let calls = tape
3530 .take()
3531 .chunks(1)
3532 .filter(|c| c[0] == b'E')
3533 .count();
3534 // Every explode the sink DID get is a commit or a tag; and a pack
3535 // with any blob or tree at all must have been spared some.
3536 let non_commit = w
3537 .entries
3538 .iter()
3539 .filter(|e| matches!(e.obj_type, ObjType::Blob | ObjType::Tree))
3540 .count();
3541 if non_commit > 0 {
3542 assert!(
3543 calls < w.entries.len(),
3544 "{name}: the sink was handed all {} objects although it wanted only commits",
3545 w.entries.len()
3546 );
3547 }
3548 }
3549 }
3550
3551 /// **A pack whose delta shape is CHOSEN, not found.** `roots` independent
3552 /// whole blobs, each the root of a binary tree of `depth` levels of
3553 /// `OFS_DELTA`s, every object exactly `body_bytes` long.
3554 ///
3555 /// The shape is the whole point of the guard below: `roots` roots is what
3556 /// gives the gatling fan-out `roots` independent units to hand four workers
3557 /// (`run_round` takes `min(workers, roots)`), the branching is what keeps a
3558 /// frame live while a deeper subtree runs — a base whose last child has not
3559 /// been reached yet is held — and one uniform object size is what makes
3560 /// "the largest single base" a number this source states rather than one a
3561 /// found pack happens to have.
3562 fn base_ceiling_pack(roots: usize, depth: u32, body_bytes: usize) -> Vec<u8> {
3563 use crate::pack_fixture::{build_pack, Item};
3564 // splitmix64, four lines, so the pack is byte-identical on every box:
3565 // incompressible bodies, so the pack's shape is the constants above and
3566 // not deflate's opinion of them.
3567 let mut state = 0x5eed_0000_0000_0001u64;
3568 let mut body = |n: usize| -> Vec<u8> {
3569 let mut v = Vec::with_capacity(n + 8);
3570 while v.len() < n {
3571 state = state.wrapping_add(0x9e37_79b9_7f4a_7c15);
3572 let mut z = state;
3573 z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
3574 z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
3575 v.extend_from_slice(&(z ^ (z >> 31)).to_le_bytes());
3576 }
3577 v.truncate(n);
3578 v
3579 };
3580 let mut items: Vec<Item> = Vec::new();
3581 for _ in 0..roots {
3582 let mut level = vec![items.len()];
3583 items.push(Item::Whole(GitObjectKind::Blob, body(body_bytes)));
3584 for _ in 1..depth {
3585 let mut next = Vec::with_capacity(level.len() * 2);
3586 for base in level {
3587 for _ in 0..2 {
3588 next.push(items.len());
3589 items.push(Item::OfsDelta { base, body: body(body_bytes) });
3590 }
3591 }
3592 level = next;
3593 }
3594 }
3595 build_pack(&items)
3596 }
3597
3598 /// **N workers get N walks, not N budgets.** The held-base ceiling is one
3599 /// atomic across every worker ([`workers_from_env`]'s doc says so in as many
3600 /// words: *"it is not N base budgets either"*), so four workers on the same
3601 /// pack hold what one budget allows plus what the walk structurally cannot
3602 /// drop — and nowhere near the four budgets a per-thread ceiling would give
3603 /// them.
3604 ///
3605 /// # 🔴 The two defects this replaces, both measured 2026-09-08
3606 ///
3607 /// It read `corpora(4)`, i.e. **whatever `.pack` files sat under
3608 /// `~/git` in `read_dir` order**, and asserted
3609 /// `peak <= budget + biggest` with
3610 ///
3611 /// ```text
3612 /// let biggest = w.entries.iter().map(|e| e.uncompressed_size).max()
3613 /// ```
3614 ///
3615 /// ```text
3616 /// ~/git/Njord/.git/objects/pack/pack-7edf01cd….pack:
3617 /// arm 1 held 267014 bytes against a 65536 budget and a 176423-byte largest object
3618 /// ```
3619 ///
3620 /// * **`biggest` was not the largest base.** `PackEntry::uncompressed_size`
3621 /// is the size field of the *pack entry*, and for an `OFS_DELTA` that is
3622 /// the length of the **delta stream**, not of the object it reconstructs.
3623 /// The peak is a sum of *resolved base payloads*, so the old assertion
3624 /// compared a peak in one unit against a maximum in another and was
3625 /// simply not about the same bytes. The sibling guard
3626 /// [`a_bounded_base_set_costs_re_inflations_not_rows`] already had this
3627 /// right: it takes the maximum over the **resolved rows** whose offset is
3628 /// used as a base, and that is what this one takes now.
3629 /// * **One base per worker, not one per pack.** `enforce_budget` never
3630 /// drops a worker's *deepest* frame — it is the base the very next child
3631 /// reads, and dropping it would re-inflate it on the following line — so
3632 /// the floor under the ceiling is one held base **per walk**, and there
3633 /// are `workers` walks. `budget + 1 × largest` was the serial arm's bound
3634 /// applied to a four-worker arm. The bound that is actually promised is
3635 /// `budget + workers × largest`: the shared ceiling, plus at most one
3636 /// undroppable base per worker (which also absorbs the only race there
3637 /// is — `live_path` is a relaxed atomic, so the peak one worker records
3638 /// may include another worker's push that has not reached its own
3639 /// `enforce_budget` yet).
3640 ///
3641 /// Neither correction weakens the guard, because the guard's *point* is the
3642 /// last assertion: `budget + workers × largest` is far below `workers ×
3643 /// budget`, which is what a per-thread ceiling would have allowed. The
3644 /// external corpus is gone with them — a test that needs another checkout
3645 /// on the same disk is not a test, and the pack is built here instead, sized
3646 /// so the ceiling demonstrably bites (`evictions > 0`, and the unbounded
3647 /// pass holds more than the bounded one).
3648 #[test]
3649 fn four_workers_hold_no_more_bases_than_the_one_budget_allows() {
3650 const ROOTS: usize = 4;
3651 const DEPTH: u32 = 8;
3652 const OBJECT: usize = 4 * 1024;
3653 // 4 roots × (2^8 − 1) = 1020 objects of 4 KiB. One walk holds at most
3654 // the seven bases above a leaf, so it wants 28 KiB and four walks want
3655 // 112 KiB — while the budget is 20 KiB: more than `workers × largest`
3656 // (16 KiB, so what binds is the ceiling and not the floor the walk
3657 // cannot drop) and less than even ONE walk wants (so it binds on the
3658 // serial arm too, and that arm is not a free pass).
3659 const BUDGET: u64 = 20 * 1024;
3660 let pack = base_ceiling_pack(ROOTS, DEPTH, OBJECT);
3661 let w = walk(&pack, 20).unwrap();
3662
3663 // Which offsets are used as a base, so "the largest base" is a maximum
3664 // over the objects the ceiling actually holds — resolved sizes, off the
3665 // rows, not pack-entry size fields.
3666 let mut is_base: std::collections::HashSet<u64> = std::collections::HashSet::new();
3667 for e in &w.entries {
3668 if let DeltaBase::Offset(o) = e.delta_base {
3669 is_base.insert(o);
3670 }
3671 }
3672
3673 let mut four_worker_peak = 0u64;
3674 for workers in [1usize, 4] {
3675 // The unbounded pass **at the same worker count**, because that is
3676 // the only thing the bounded one is comparable with: N walks hold N
3677 // paths whether or not there is a ceiling, so a one-worker baseline
3678 // would make the four-worker arm look like a leak.
3679 let free_limits = ResolveLimits { workers, ..ResolveLimits::none() };
3680 let (free, free_report, _) = stream(&pack, free_limits);
3681 assert_eq!(free_report.evictions, 0, "{workers} workers: the unbounded pass evicted");
3682 let largest = free
3683 .iter()
3684 .filter(|r| is_base.contains(&r.offset))
3685 .map(|r| r.uncompressed_size)
3686 .max()
3687 .expect("the pack has bases");
3688 assert_eq!(largest, OBJECT as u64, "every object in this pack is one size");
3689
3690 let limits = ResolveLimits { workers, base_budget: BUDGET, ..ResolveLimits::none() };
3691 let mut rows = Vec::new();
3692 let report = resolve_streaming(
3693 &pack,
3694 &w,
3695 GitHashKind::Sha1,
3696 0,
3697 &NoBases,
3698 &NoSink,
3699 limits,
3700 &mut |r| {
3701 rows.extend(r.drain(..));
3702 Ok(())
3703 },
3704 )
3705 .expect("resolve");
3706 assert_eq!(report.workers as usize, workers, "the fan-out took the worker count");
3707 assert_eq!(
3708 rows_of(&free),
3709 rows_of(&rows),
3710 "{workers} workers: the ceiling changed the output"
3711 );
3712 assert!(
3713 report.evictions > 0,
3714 "{workers} workers: nothing was evicted, so the ceiling never bit and this \
3715 guard proves nothing"
3716 );
3717 assert!(
3718 free_report.peak_base_bytes > report.peak_base_bytes,
3719 "{workers} workers: unbounded held {} and bounded {} — the ceiling did nothing",
3720 free_report.peak_base_bytes,
3721 report.peak_base_bytes
3722 );
3723 let allowed = BUDGET + workers as u64 * largest;
3724 assert!(
3725 report.peak_base_bytes <= allowed,
3726 "{workers} workers: held {} bytes against a {BUDGET}-byte budget plus {workers} × \
3727 the {largest}-byte largest base = {allowed}",
3728 report.peak_base_bytes
3729 );
3730 if workers == 4 {
3731 four_worker_peak = report.peak_base_bytes;
3732 // The whole claim, stated as the number a PER-THREAD ceiling
3733 // would have produced: four walks each allowed the budget plus
3734 // their own undroppable base.
3735 let per_thread = workers as u64 * (BUDGET + largest);
3736 assert!(
3737 four_worker_peak < per_thread,
3738 "four workers held {four_worker_peak} bytes, which is four ceilings of \
3739 {BUDGET} + {largest} = {per_thread}: the budget is per thread, not per pack"
3740 );
3741 }
3742 eprintln!(
3743 "{workers} worker(s): peak {} B (unbounded {} B), budget {BUDGET} B, largest base \
3744 {largest} B, {} evictions, {} re-inflations",
3745 report.peak_base_bytes,
3746 free_report.peak_base_bytes,
3747 report.evictions,
3748 report.reinflations
3749 );
3750 }
3751 }
3752
3753 /// **The chain memo answers the same bytes as the chain, and a hit walks
3754 /// nothing.** A whole tree, a delta on it and a delta on that: read the
3755 /// deepest one through a memo twice. The second read is a hit at its own
3756 /// offset (depth 0), and reading the middle one afterwards is a hit too —
3757 /// every link applied on the way down was put in the memo. The bytes are
3758 /// the uncached answer's, byte for byte.
3759 #[test]
3760 fn a_chain_memo_answers_the_chain_and_a_hit_walks_nothing() {
3761 use crate::pack_fixture::{Item, build_pack};
3762 let base = b"100644 a\0".iter().copied().chain([7u8; 20]).collect::<Vec<u8>>();
3763 let mut mid = base.clone();
3764 mid.extend_from_slice(b"100644 b\0");
3765 mid.extend_from_slice(&[9u8; 20]);
3766 let mut top = mid.clone();
3767 top.extend_from_slice(b"100644 c\0");
3768 top.extend_from_slice(&[11u8; 20]);
3769 let pack = build_pack(&[
3770 Item::Whole(GitObjectKind::Tree, base.clone()),
3771 Item::OfsDelta { base: 0, body: mid.clone() },
3772 Item::OfsDelta { base: 1, body: top.clone() },
3773 ]);
3774 let w = crate::pack_walk::walk(&pack, 20).unwrap();
3775 let archive = SliceArchive(&pack);
3776 let none = |_: &[u8]| None;
3777 let top_at = w.entries[2].offset;
3778 let mid_at = w.entries[1].offset;
3779
3780 let (k0, plain, cost0) = materialize_at(&archive, top_at, 20, &none, &NoBases).unwrap();
3781 assert_eq!((k0, plain.as_slice(), cost0.depth), (GitObjectKind::Tree, top.as_slice(), 2));
3782
3783 let mut memo = ChainCache::new(1 << 20);
3784 let (k1, first, cost1) =
3785 materialize_at_with(&archive, top_at, 20, &none, &NoBases, Some(&mut memo)).unwrap();
3786 assert_eq!((k1, first.as_slice(), cost1.depth), (GitObjectKind::Tree, top.as_slice(), 2), "a cold memo is the chain");
3787 let (k2, again, cost2) =
3788 materialize_at_with(&archive, top_at, 20, &none, &NoBases, Some(&mut memo)).unwrap();
3789 assert_eq!((k2, again.as_slice()), (GitObjectKind::Tree, top.as_slice()));
3790 assert_eq!(cost2.depth, 0, "a hit at the asked offset walks no link");
3791 let (k3, middle, cost3) =
3792 materialize_at_with(&archive, mid_at, 20, &none, &NoBases, Some(&mut memo)).unwrap();
3793 assert_eq!((k3, middle.as_slice(), cost3.depth), (GitObjectKind::Tree, mid.as_slice(), 0), "every link on the way down was memoised");
3794 let (hits, misses) = memo.stats();
3795 assert_eq!((hits, misses), (2, 3), "first read: 3 misses (top, mid, base); then 2 hits");
3796 assert!(memo.bytes() == base.len() + mid.len() + top.len(), "three payloads held");
3797
3798 // Bounded: a memo too small for the base holds nothing of it and still
3799 // answers the same bytes.
3800 let mut tiny = ChainCache::new(mid.len());
3801 let (_, b1, _) = materialize_at_with(&archive, top_at, 20, &none, &NoBases, Some(&mut tiny)).unwrap();
3802 let (_, b2, _) = materialize_at_with(&archive, top_at, 20, &none, &NoBases, Some(&mut tiny)).unwrap();
3803 assert_eq!((b1.as_slice(), b2.as_slice()), (top.as_slice(), top.as_slice()));
3804 assert!(tiny.bytes() <= mid.len(), "never over its cap");
3805 }
3806}