znippy-plugin-git 0.1.1

//! **What one pack costs the heap while it is being resolved.**
//!
//! `resolve_walked` is the indexer's whole job and the only place a push
//! materialises inflated content, so its peak live heap *is* the server's push
//! footprint. This measures that peak against the one number it has to be
//! compared with — the pack's total inflated payload — because a resolver that
//! peaks at the size of the object set is holding the object set, whatever its
//! comments say.
//!
//! # Why a global allocator and not RSS
//!
//! `/proc/self/status` reports what the allocator has taken from the kernel,
//! which never comes back down: on oden a 5.9 MB pack left a `gunnar-server`
//! sitting at 575 MB of `RssAnon` and it stayed there through three clones and
//! 60 s of idle. That number cannot tell a live byte from a freed one — and the
//! distinction is the entire question. `MALLOC_ARENA_MAX=1` moved it by 0.6 MB,
//! which is how we know arena count was not the story either.
//!
//! A counting allocator answers exactly the right thing: **live bytes, peak over
//! the call**. It lives in an integration test so the `#[global_allocator]` is
//! scoped to this one test binary and no shipped code carries it.
//!
//! # MEASURED on oden, 2026-08-11
//!
//! One real pack, `resolve_walked` with [`NoSink`] so nothing but the resolver
//! is in the measurement:
//!
//! | | peak live heap | ×payload |
//! |---|---:|---:|
//! | before | 40.9 MB | **2.15x** |
//! | after | 3.5 MB | **0.18x** |
//!
//! The 2.15x was two full copies of every retained object: `content` kept one
//! and never pruned it, and `Resolved::payload` kept the other for a reader that
//! does not exist. The same shape, on the 36 172-object push fixture, is the
//! 575 MB.
//!
//! # The `SKIP` lines, and the bug they found
//!
//! Running this against the resolver as it stood before 2026-08-11 reports
//! **six of these twelve packs as unresolvable**, each with "this pack is thin:
//! it deltas against `<oid>` which is not in it". Every one of those packs is a
//! `.git/objects/pack` file that `git verify-pack` accepts, and on-disk git
//! packs are self-contained by construction — so the claim was false.
//!
//! The base it named in the first case, `d76f39fa…`, **is in that pack**: a
//! 1097-byte blob at offset 13752726. The old retention rule was
//! `needed.contains(&offset) || kind != Blob`, and `needed` was built from
//! `OFS_DELTA` bases only. A blob that is the base of a `REF_DELTA` and of no
//! `OFS_DELTA` therefore matched neither arm, was dropped the moment it was
//! hashed, and the entry that deltas against it was reported as an external
//! dependency this repository does not have.
//!
//! Counting `REF_DELTA` bases by oid — which the current resolver has to do
//! anyway, to know when to drop one — closes it. In production the store, not
//! `NoBases`, answers the second lookup, so this surfaced as a slow path
//! (`ExplodedStats::rederived`, a whole-pack re-resolve per object) rather than
//! as a failed push. It is a refusal of a valid pack all the same, and this
//! harness is what made it visible: a `continue` hid it until the `SKIP` branch
//! was made to say why.

use std::alloc::{GlobalAlloc, Layout, System};
use std::path::PathBuf;
use std::sync::atomic::{AtomicUsize, Ordering};

use znippy_plugin_git::exploded::NoSink;
use znippy_plugin_git::object::GitHashKind;
use znippy_plugin_git::pack_walk::walk;
use znippy_plugin_git::resolve::{NoBases, resolve_walked};

// ── the counting allocator ──────────────────────────────────────────────────

static LIVE: AtomicUsize = AtomicUsize::new(0);
static PEAK: AtomicUsize = AtomicUsize::new(0);

struct Counting;

impl Counting {
    fn grew(by: usize) {
        let now = LIVE.fetch_add(by, Ordering::Relaxed).saturating_add(by);
        // A plain max: the peak only ever rises, so a lost race costs at most
        // one sample and never reports a peak that did not happen.
        PEAK.fetch_max(now, Ordering::Relaxed);
    }

    /// **Saturating, and it has to be.** This was `LIVE.fetch_sub(by)` and the
    /// whole test died `attempt to add with overflow` in `grew` — because a
    /// counter that started at 0 and then saw a `dealloc` for an allocation made
    /// before it was installed wraps to near `usize::MAX`, and the next `alloc`
    /// overflows adding to it. Nothing was wrong with the resolver; the
    /// instrument was destroying its own run. Its sibling
    /// `emit_peak_memory.rs` was written saturating from the start for this
    /// reason, and this is the same fix.
    fn shrank(by: usize) {
        let _ = LIVE.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |v| {
            Some(v.saturating_sub(by))
        });
    }
}

unsafe impl GlobalAlloc for Counting {
    unsafe fn alloc(&self, l: Layout) -> *mut u8 {
        let p = unsafe { System.alloc(l) };
        if !p.is_null() {
            Self::grew(l.size());
        }
        p
    }
    unsafe fn dealloc(&self, p: *mut u8, l: Layout) {
        Self::shrank(l.size());
        unsafe { System.dealloc(p, l) }
    }
    unsafe fn realloc(&self, p: *mut u8, l: Layout, new: usize) -> *mut u8 {
        let q = unsafe { System.realloc(p, l, new) };
        if !q.is_null() {
            if new >= l.size() {
                Self::grew(new - l.size());
            } else {
                Self::shrank(l.size() - new);
            }
        }
        q
    }
    unsafe fn alloc_zeroed(&self, l: Layout) -> *mut u8 {
        let p = unsafe { System.alloc_zeroed(l) };
        if !p.is_null() {
            Self::grew(l.size());
        }
        p
    }
}

#[global_allocator]
static A: Counting = Counting;

/// Every real pack under 32 MB in a corpus root, biggest first.
///
/// Same corpus `resolve.rs`'s own gates use, and the same skip-if-absent rule:
/// this asserts a property of the resolver over real git data, and there is no
/// synthetic pack whose delta chains are worth measuring instead.
///
/// **The whole corpus and not one pack**, because the ratio is a property of a
/// pack's delta shape and the packs differ wildly in it: znippy's own pack
/// retains 0.75x (most of its blobs are nobody's base and are dropped), while a
/// linear-history pack where every blob is the next commit's base retains 2.15x.
/// One pack would have been a gate that passed by luck.
fn real_packs() -> Vec<PathBuf> {
    let mut out: Vec<(u64, PathBuf)> = Vec::new();
    for root in [
        "/home/rickard/git",
        "/home/rickard/scratch/gunnar-bench-fixtures",
    ] {
        let Ok(repos) = std::fs::read_dir(root) else {
            continue;
        };
        for repo in repos.flatten() {
            for sub in [".git/objects/pack", "objects/pack"] {
                let Ok(files) = std::fs::read_dir(repo.path().join(sub)) else {
                    continue;
                };
                for f in files.flatten() {
                    let p = f.path();
                    if p.extension().is_some_and(|e| e == "pack") {
                        let len = f.metadata().map(|m| m.len()).unwrap_or(0);
                        if len < (32 << 20) {
                            out.push((len, p));
                        }
                    }
                }
            }
        }
    }
    out.sort_by_key(|(len, _)| std::cmp::Reverse(*len));
    out.into_iter().map(|(_, p)| p).collect()
}

/// **The resolver must not hold the object set it is producing.**
///
/// Seen RED on the code as it stood: 40.9 MB peak against 19.0 MB of payload,
/// **2.15x** — `content` held every kept object for the whole call (nothing ever
/// removed a row, despite the comment saying "held only while something still
/// needs them") and `Resolved::payload` held a second copy of the same bytes.
///
/// The bound is stated against the **payload**, not against a constant: a
/// resolver whose peak scales with the object set fails it on any pack, and one
/// that keeps only its live delta bases passes it on any pack. `0.75x` leaves
/// generous room for the largest single base chain and for the walk's own
/// entry vector, while still being unreachable for anything that retains
/// everything.
#[test]
fn resolving_a_pack_does_not_hold_the_whole_object_set() {
    let packs = real_packs();
    if packs.is_empty() {
        eprintln!("no real pack on this machine; nothing to measure");
        return;
    }
    let mut measured = 0usize;
    let mut worst: Option<(f64, String)> = None;
    // Reported, never asserted. A timing assertion on a shared box is a coin
    // toss dressed as a gate; this number exists so the same test binary can be
    // run against two revisions of `src/resolve.rs` and the CPU cost of the
    // change read off directly. That is what settled whether the reference
    // counting was a CPU regression, and it is not: over the 6 packs both
    // revisions accept, 1.058 s before against 0.975 s after — ~8 % faster.
    //
    // Compare only the packs BOTH revisions accept. The totals are not
    // comparable on their own, because the older resolver refuses half this
    // corpus (see the `SKIP` lines and the module docs).
    let mut total_secs = 0.0f64;
    let mut total_payload = 0u64;
    for pack_path in packs.iter().take(12) {
        let Ok(pack) = std::fs::read(pack_path) else {
            continue;
        };
        // A thin pack cannot be resolved without its store, and there is no
        // store here — skip it rather than report a resolver failure as a
        // memory result.
        let Ok(w) = walk(&pack, 20) else { continue };

        // Everything above is setup and must not be in the measurement.
        LIVE.store(0, Ordering::Relaxed);
        PEAK.store(0, Ordering::Relaxed);
        let t0 = std::time::Instant::now();
        // A pack that will not resolve is REPORTED, never silently skipped. A
        // bare `else { continue }` here hid the most interesting thing this
        // harness ever found: the resolver used to refuse six of these twelve
        // packs outright (see the module docs).
        let rows = match resolve_walked(&pack, &w, GitHashKind::Sha1, 0, &NoBases, &NoSink) {
            Ok(rows) => rows,
            Err(e) => {
                eprintln!("SKIP  {}\n        {e}", pack_path.display());
                continue;
            }
        };
        let elapsed = t0.elapsed();
        let peak = PEAK.load(Ordering::Relaxed);
        total_secs += elapsed.as_secs_f64();

        let payload: u64 = rows.iter().map(|r| r.uncompressed_size).sum();
        if payload < 4_000_000 {
            continue;
        }
        let ratio = peak as f64 / payload as f64;
        eprintln!(
            "{:.2}x  peak {:>7.1} MB / payload {:>7.1} MB  {:>6} objects  {:>7.3}s  {}",
            ratio,
            peak as f64 / 1e6,
            payload as f64 / 1e6,
            rows.len(),
            elapsed.as_secs_f64(),
            pack_path.display(),
        );
        measured += 1;
        total_payload += payload;
        if worst.as_ref().is_none_or(|(w, _)| ratio > *w) {
            worst = Some((ratio, pack_path.display().to_string()));
        }
    }
    eprintln!(
        "TOTAL {measured} packs, {:.1} MB inflated payload, {total_secs:.3}s in resolve_walked \
         ({:.1} MB/s)",
        total_payload as f64 / 1e6,
        total_payload as f64 / 1e6 / total_secs,
    );
    assert!(
        measured >= 3,
        "only {measured} pack(s) were big enough to measure — too few to prove anything about \
         scaling"
    );
    let (ratio, which) = worst.expect("measured >= 3 means there is a worst");
    assert!(
        ratio < 0.75,
        "resolving {which} peaked at {ratio:.2}x its own inflated payload. The resolver is \
         holding the object set, not a delta-base working set."
    );
}