znippy-plugin-git 0.1.1

//! **Is the 4.8 GB plateau REACHABLE, or UNREACHABLE-BUT-UNFREED?**
//!
//! # Why this exists before any more eviction work
//!
//! gunnar-on-znippy sits at 4792–4872 MB of anonymous RSS on oden's forge suite,
//! **flat** — the same figure whether the clone it served was 3 257 objects or 352 396.
//! Two rounds of work have treated that as a retention bug and bounded a cache; the
//! second round (`gunnar` `ecc2c842`) reported its own fix as failed, correctly: the
//! bound is `DEFAULT_OPEN_REPOS = 256` and the forge seeds ~20 repositories, so it never
//! fires.
//!
//! **An eviction bound is a tool for a growth curve.** Against a flat plateau it can only
//! help if the plateau exceeds the bound. So the question that has to be answered before
//! any more dropping is which of two completely different defects this is:
//!
//! * **(a) REACHABLE** — something deliberately holds those bytes. The fix is to find the
//!   owner and decide whether it should hold them.
//! * **(b) UNREACHABLE BUT UNFREED** — the bytes are `free()`d and the allocator never
//!   hands the pages back to the OS. **No amount of dropping helps**; the fix is an
//!   allocator or arena change.
//!
//! Every measurement taken so far — RSS while the server runs — is identical under both,
//! which is why three sessions could not tell them apart.
//!
//! # The discriminator
//!
//! For each phase, three readings of anonymous RSS out of `/proc/self/smaps_rollup`
//! (a file read; no shelling, LAW):
//!
//! 1. **live** — the structure built and warm.
//! 2. **dropped** — every owner gone. Anything still resident is now, by definition,
//!    unreachable from the program.
//! 3. **trimmed** — after `malloc_trim(0)`, which asks glibc to return free arena pages.
//!
//! | live → dropped | dropped → trimmed | verdict |
//! |---|---|---|
//! | falls | — | the drop works; the plateau's owner is upstream of this structure |
//! | flat | **falls** | **(b)** glibc retention — eviction cannot fix it |
//! | flat | flat | **(a)** something still holds it, or the arena is too fragmented to trim |
//!
//! # The two controls are the point, not decoration
//!
//! glibc routes allocations at or above `M_MMAP_THRESHOLD` (128 KiB by default) straight
//! to `mmap`, and `free` on those `munmap`s immediately — those pages always go back.
//! Below it, allocations come from the arena and `free` returns them to a free list, not
//! to the OS. So *the same number of bytes behaves completely differently depending on
//! the block size it was requested in*, and a probe that does not pin that down cannot
//! attribute what it sees.
//!
//! That matters here specifically: redb's page cache holds **4 KiB pages**, far below the
//! threshold. 64 MiB of cache is ~16 000 small blocks, and freeing them is exactly the
//! shape that stays in the arena. The controls measure that directly on this box rather
//! than quoting it from documentation.
//!
//! # Run
//!
//! ```bash
//! cargo test -p znippy-plugin-git --test rss_retention_probe -- --ignored --nocapture
//! ```
//!
//! `#[ignore]` because it is a measurement, it allocates ~1 GB, and libtest would
//! otherwise run it beside other tests in the same process — where their allocations land
//! in the same arena and the readings stop meaning anything.

use std::path::Path;

use znippy_plugin_git::arms::DEFAULT_REDB_CACHE_BYTES;
// `lookup` reaches the stack through the `ObjectIndex` trait it implements, not an
// inherent method — the trait has to be in scope for the cache-warming reads.
use znippy_plugin_git::index_layout::{IndexEntry, ObjType, ObjectIndex, OneTableFourColumns};
use znippy_plugin_git::read_stack::{ObjectReadStack, RebuildTriggers};

/// Anonymous RSS of THIS process, in KiB, from `/proc/self/smaps_rollup`.
///
/// `Anonymous:` and not `Rss:` deliberately. `Rss` includes file-backed pages — mapped
/// binaries, and any page cache a mapping brings in — which the kernel reclaims on demand
/// and which no allocator change would touch. The plateau under investigation was already
/// reported as anonymous, and this keeps the two from being conflated.
fn anon_rss_kb() -> u64 {
    let s = std::fs::read_to_string("/proc/self/smaps_rollup")
        .expect("/proc/self/smaps_rollup (Linux 4.14+)");
    for line in s.lines() {
        if let Some(rest) = line.strip_prefix("Anonymous:") {
            let kb = rest.trim().trim_end_matches("kB").trim();
            return kb.parse().expect("Anonymous: is a kB count");
        }
    }
    panic!("no `Anonymous:` line in smaps_rollup");
}

fn mb(kb: u64) -> f64 {
    kb as f64 / 1024.0
}

/// **The allocator's own books**: `(in_use, from_kernel)` in KiB, via `mallinfo2`.
///
/// This is the reading that settles reachable-vs-unfreed **without dropping anything**,
/// and it is why the probe leads with it. `uordblks + hblkhd` is what the program has
/// asked for and not given back — *live, reachable bytes*. `arena + hblkhd` is what
/// glibc has taken from the kernel. The gap between them is free list: bytes the program
/// no longer owns and the OS has not got back.
///
/// Three sessions measured RSS, which is the SUM of those two and therefore cannot
/// distinguish them. The technique is not new here — `znippy-zoomies`'
/// `osm-katana/src/phase_log.rs` reached for exactly this pair on 2026-08-03 for the same
/// question, with the same reasoning written down ("a number near zero says the memory is
/// genuinely live … a large number says the allocator was the leak"). It is read here
/// rather than imported because `osm-katana` sits DOWNSTREAM of the `znippy-zoomies` root
/// that znippy depends on, so importing it would invert the dependency; see the report's
/// note on where the shared primitive belongs.
fn mallinfo_kb() -> Option<(u64, u64)> {
    #[cfg(all(target_os = "linux", target_env = "gnu"))]
    {
        // SAFETY: `mallinfo2` takes no arguments and returns a plain POD struct.
        let mi = unsafe { libc::mallinfo2() };
        let in_use = (mi.uordblks as u64 + mi.hblkhd as u64) / 1024;
        let from_kernel = (mi.arena as u64 + mi.hblkhd as u64) / 1024;
        return Some((in_use, from_kernel));
    }
    #[cfg(not(all(target_os = "linux", target_env = "gnu")))]
    None
}

fn mallinfo_line(tag: &str) -> String {
    match mallinfo_kb() {
        Some((used, from_kernel)) => format!(
            "{tag}: in-use {:8.1} MB · from-kernel {:8.1} MB · free-list {:8.1} MB",
            mb(used),
            mb(from_kernel),
            mb(from_kernel.saturating_sub(used))
        ),
        None => format!("{tag}: mallinfo2 unavailable (not glibc)"),
    }
}

/// Ask glibc to return free arena pages to the OS. Safe to call at any time; it is a
/// hint, and on a non-glibc libc the symbol would not link — which is itself the honest
/// signal that this whole diagnosis is glibc-specific.
fn malloc_trim() {
    unsafe { libc::malloc_trim(0) };
}

/// One phase's three readings, printed as the table the report quotes.
struct Phase {
    what: String,
    base: u64,
    live: u64,
    dropped: u64,
    trimmed: u64,
}

impl Phase {
    fn report(&self) -> Self {
        let grew = self.live.saturating_sub(self.base);
        let after_drop = self.dropped.saturating_sub(self.base);
        let after_trim = self.trimmed.saturating_sub(self.base);
        eprintln!(
            "\n── {} ──\n  grew            {:8.1} MB\n  after drop      {:8.1} MB  ({:5.1}% still \
             resident)\n  after trim      {:8.1} MB  ({:5.1}% still resident)",
            self.what,
            mb(grew),
            mb(after_drop),
            100.0 * after_drop as f64 / grew.max(1) as f64,
            mb(after_trim),
            100.0 * after_trim as f64 / grew.max(1) as f64,
        );
        Self { what: self.what.clone(), ..*self }
    }
    fn held_after_drop_pct(&self) -> f64 {
        let grew = self.live.saturating_sub(self.base).max(1);
        100.0 * self.dropped.saturating_sub(self.base) as f64 / grew as f64
    }
    fn held_after_trim_pct(&self) -> f64 {
        let grew = self.live.saturating_sub(self.base).max(1);
        100.0 * self.trimmed.saturating_sub(self.base) as f64 / grew as f64
    }
}

/// **CONTROL A — many SMALL blocks**, the shape redb's 4 KiB page cache has.
/// Expectation, to be confirmed rather than assumed: the drop frees nothing back to the
/// OS and `malloc_trim` is what returns it.
fn control_small_blocks(target_mb: usize) -> Phase {
    const BLOCK: usize = 4096;
    let n = target_mb * 1024 * 1024 / BLOCK;
    let base = anon_rss_kb();
    let mut v: Vec<Vec<u8>> = Vec::with_capacity(n);
    for i in 0..n {
        // Written, not just reserved: an untouched page is not resident, and a probe
        // that measured reservations would report a number the kernel never backed.
        let mut b = vec![0u8; BLOCK];
        b[0] = i as u8;
        b[BLOCK - 1] = (i >> 8) as u8;
        v.push(b);
    }
    let live = anon_rss_kb();
    drop(v);
    let dropped = anon_rss_kb();
    malloc_trim();
    let trimmed = anon_rss_kb();
    Phase { what: format!("CONTROL A — {n} x 4 KiB blocks (redb page shape)"), base, live, dropped, trimmed }
}

/// **CONTROL B — few LARGE blocks**, above glibc's mmap threshold. The counterweight: if
/// this one also failed to return its pages, the probe would be measuring something other
/// than the arena and Control A would prove nothing.
fn control_large_blocks(target_mb: usize) -> Phase {
    const BLOCK: usize = 1024 * 1024;
    let n = target_mb;
    let base = anon_rss_kb();
    let mut v: Vec<Vec<u8>> = Vec::with_capacity(n);
    for i in 0..n {
        let mut b = vec![0u8; BLOCK];
        for p in (0..BLOCK).step_by(4096) {
            b[p] = i as u8;
        }
        v.push(b);
    }
    let live = anon_rss_kb();
    drop(v);
    let dropped = anon_rss_kb();
    malloc_trim();
    let trimmed = anon_rss_kb();
    Phase { what: format!("CONTROL B — {n} x 1 MiB blocks (above mmap threshold)"), base, live, dropped, trimmed }
}

/// The per-database redb page-cache ceiling this run uses. Defaults to the shipped
/// [`DEFAULT_REDB_CACHE_BYTES`]; override with `PROBE_REDB_CACHE_BYTES` **as a plain byte
/// count** to measure what the ceiling is actually worth.
///
/// Deliberately a *separate* variable from the product's own `ZNIPPY_GIT_REDB_CACHE_BYTES`,
/// and deliberately parsed the same strict way: `arms::redb_cache_bytes()` does
/// `raw.parse::<usize>()` and **`bail!`s on anything that is not a bare integer** — so
/// `=8m` is an error, never 8 MiB. Its own doc says why, in writing: *"a silent fallback
/// would let an operator believe a measurement came from a ceiling that was never
/// applied."* A probe that took `8m` here and reported a number would be that operator.
fn probe_cache_bytes() -> usize {
    match std::env::var("PROBE_REDB_CACHE_BYTES") {
        Ok(v) if !v.trim().is_empty() => v.trim().parse::<usize>().unwrap_or_else(|_| {
            panic!(
                "PROBE_REDB_CACHE_BYTES={v:?} is not a bare byte count. Suffixes are NOT \
                 parsed — write 8388608, not 8m. Failing loudly rather than measuring a \
                 ceiling that was never applied."
            )
        }),
        _ => DEFAULT_REDB_CACHE_BYTES,
    }
}

/// Where the probe writes its redb fixtures.
///
/// **`/tmp` on oden is tmpfs — RAM.** A memory probe that puts a gigabyte of databases in
/// `std::env::temp_dir()` there is charging the box for its own fixtures, and the harness
/// on this machine had already leaked 96 GB into `/tmp` as abandoned store dirs before
/// anyone noticed. Set `PROBE_SCRATCH` to a real disk (md1). If the chosen directory is
/// tmpfs this says so out loud rather than quietly measuring itself — the fixtures land in
/// page cache and not in `Anonymous:`, so they do not corrupt THIS reading, but they do
/// consume the memory the probe exists to reason about.
fn scratch_root() -> std::path::PathBuf {
    let root = std::env::var("PROBE_SCRATCH")
        .ok()
        .filter(|s| !s.trim().is_empty())
        .map(std::path::PathBuf::from)
        .unwrap_or_else(std::env::temp_dir);
    if is_tmpfs(&root) {
        eprintln!(
            "  WARNING: {} is tmpfs (RAM). The redb fixtures for this run are being written \
             into memory. Set PROBE_SCRATCH to a real disk.",
            root.display()
        );
    }
    root
}

/// Is `p` under a tmpfs mount? Read out of `/proc/mounts` — a file read, no shelling.
/// Longest matching mount point wins, so `/tmp` beats `/`.
fn is_tmpfs(p: &Path) -> bool {
    let Ok(mounts) = std::fs::read_to_string("/proc/mounts") else { return false };
    let mut best: Option<(usize, bool)> = None;
    for line in mounts.lines() {
        let mut f = line.split_whitespace();
        let (_dev, point, fstype) = (f.next(), f.next(), f.next());
        let (Some(point), Some(fstype)) = (point, fstype) else { continue };
        if p.starts_with(point) {
            let len = point.len();
            if best.is_none_or(|(b, _)| len > b) {
                best = Some((len, fstype == "tmpfs"));
            }
        }
    }
    best.is_some_and(|(_, t)| t)
}

fn synthetic(n: usize, seed: u64) -> Vec<IndexEntry> {
    let mut s = seed | 1;
    let mut out = Vec::with_capacity(n);
    let mut offset = 12u64;
    for _ in 0..n {
        // xorshift — deterministic, no rand dep, and the oids must be well spread or
        // the redb btree degenerates and the page count stops resembling a real store.
        s ^= s << 13;
        s ^= s >> 7;
        s ^= s << 17;
        let mut oid = vec![0u8; 20];
        oid[..8].copy_from_slice(&s.to_le_bytes());
        oid[8..16].copy_from_slice(&s.rotate_left(21).to_le_bytes());
        let len = 64 + (s % 4096);
        out.push(IndexEntry {
            oid,
            offset,
            len,
            obj_type: ObjType::Blob,
            uncompressed_size: len * 3,
            delta_base: 0,
        });
        offset += len;
    }
    out
}

/// **THE REAL PATH — `ObjectReadStack` over redb, at the forge's shape.**
///
/// `repos` stacks, each carrying `rows` entries, each opened with the SAME
/// `DEFAULT_REDB_CACHE_BYTES` the server uses. The cache is deliberately left at its
/// default: shrinking it is a different experiment (one already queued elsewhere), and it
/// answers "is the cache the owner", not "is the plateau reachable" — which is this
/// question and has to be settled first.
fn real_stacks(dir: &Path, repos: usize, rows: usize) -> Phase {
    let base = anon_rss_kb();
    let mut stacks = Vec::with_capacity(repos);
    for r in 0..repos {
        let path = dir.join(format!("repo{r}.objects.tail.redb"));
        let stack = ObjectReadStack::<OneTableFourColumns>::open(
            &path,
            RebuildTriggers::manual(),
            probe_cache_bytes(),
        )
        .expect("open a read stack");
        let entries = synthetic(rows, 0x9E3779B97F4A7C15 ^ (r as u64));
        stack.append(&entries).expect("append");
        stack.rebuild().expect("rebuild");
        // WARM THE CACHE. Without reads the redb page cache holds almost nothing and the
        // probe would measure an empty cache and conclude the cache is small — the exact
        // way a memory probe lies.
        for e in entries.iter().step_by(7) {
            let _ = stack.lookup(&e.oid);
        }
        stacks.push(stack);
    }
    let live = anon_rss_kb();
    let held_mb = mb(live.saturating_sub(base));
    eprintln!(
        "  [live] {repos} stacks x {rows} rows = {:.1} MB anonymous, {:.1} MB per stack \
         (cache cap {} MiB/db)",
        held_mb,
        held_mb / repos as f64,
        probe_cache_bytes() / (1024 * 1024),
    );
    // THE READING THAT ANSWERS (a) WITHOUT DROPPING ANYTHING. If in-use is close to the
    // resident growth here, these bytes are genuinely live and reachable while the
    // handles are open — which is the forge's actual state, because its LRU bound (256)
    // never fires over ~20 repositories and nothing is ever released.
    eprintln!("  {}", mallinfo_line("[live]   "));
    drop(stacks);
    let dropped = anon_rss_kb();
    eprintln!("  {}", mallinfo_line("[dropped]"));
    malloc_trim();
    let trimmed = anon_rss_kb();
    eprintln!("  {}", mallinfo_line("[trimmed]"));
    Phase {
        what: format!("REAL — {repos} x ObjectReadStack over redb ({rows} rows each)"),
        base,
        live,
        dropped,
        trimmed,
    }
}

#[test]
#[ignore = "measurement: allocates ~1 GB and must not share a process with other tests"]
fn reachable_or_merely_unfreed() {
    eprintln!(
        "\n=== RSS RETENTION PROBE ===\nallocator: {} (no #[global_allocator] in \
         gunnar-server or znippy ⇒ system malloc)\n",
        if cfg!(target_env = "gnu") { "glibc malloc" } else { "NOT glibc — read the verdict with care" }
    );

    // ORDER MATTERS, AND GETTING IT WRONG COST A RUN. On the first pass the small-block
    // control ran FIRST, and both controls then held ~99% after drop — the guard below
    // fired and refused to attribute anything, correctly. The cause is not exotic: the
    // small control frees ~700 MB into the arena, `malloc_trim` returns those pages to
    // the kernel but the free CHUNKS remain in glibc's books, and the large control's
    // 1 MiB requests are then satisfied out of that free list instead of by `mmap`. A
    // control that never reached the mmap path could not measure the mmap path.
    //
    // Large first, on a fresh arena, is the only order in which the two are independent.
    let b = control_large_blocks(700).report();
    let a = control_small_blocks(700).report();

    let dir = tempfile::Builder::new()
        .prefix("znippy-rss-probe-")
        .tempdir_in(scratch_root())
        .expect("tempdir on the scratch root");
    let real = real_stacks(dir.path(), 20, 300_000).report();

    eprintln!("\n=== VERDICT ===");

    // ── THE CALIBRATION, and it is not the one this probe was first written with ──────
    //
    // The original guard here asserted the two controls would DISAGREE — small blocks
    // held in the arena, large blocks `munmap`ped straight back. **That premise is false
    // on this box**: with a fresh arena BOTH controls return ~100% of their bytes on
    // `drop` alone, before any trim. Recorded as a wrong prediction rather than quietly
    // deleted, because it is load-bearing: it means block size is NOT what separates the
    // controls from the real workload, so the real workload's retention needs a different
    // explanation (fragmentation across live allocations and per-thread arenas, which
    // top-of-heap trimming cannot reach — see the report).
    //
    // What the controls DO establish, and it is the only thing this probe needs from
    // them: **the instrument can see memory come back.** A probe that could not would
    // report "still resident" for everything and prove nothing. Both controls freeing
    // clean is that calibration.
    for c in [&b, &a] {
        assert!(
            c.held_after_drop_pct() < 10.0,
            "{}: a control that allocates and frees a known {:.0} MB left {:.1}% resident. \
             This probe cannot detect memory being returned, so NOTHING below can be \
             attributed — the reading is about the instrument, not the workload.",
            c.what,
            mb(c.live.saturating_sub(c.base)),
            c.held_after_drop_pct()
        );
    }
    eprintln!(
        "instrument OK: both controls returned their bytes on drop ({:.1}% / {:.1}% left), \
         so a return IS visible to this probe.",
        b.held_after_drop_pct(),
        a.held_after_drop_pct()
    );

    let held_drop = real.held_after_drop_pct();
    let held_trim = real.held_after_trim_pct();
    // ── (a) — asked of the LIVE reading, which is the forge's actual state ────────────
    //
    // The forge's LRU bound is DEFAULT_OPEN_REPOS = 256 against ~20 repositories, so it
    // never fires and NOTHING IS EVER RELEASED there. The live in-use figure is therefore
    // the one that describes production, and the drop/trim pair below describes what
    // would happen if eviction were fixed.
    eprintln!(
        "(a) while LIVE: {:.1} MB resident for {} databases — see the [live] in-use line \
         above. If in-use tracks resident, these bytes are REACHABLE and deliberately \
         held, and the owner is the per-database redb page cache at its \
         {} MiB cap.",
        mb(real.live.saturating_sub(real.base)),
        20,
        probe_cache_bytes() / (1024 * 1024),
    );

    // ── (b) — asked of the drop/trim pair ────────────────────────────────────────────
    if held_drop < 30.0 {
        eprintln!(
            "(b) NO: dropping the stacks returned {:.1}% on its own, so releasing a store \
             does give RSS back and eviction alone would be a complete fix.",
            100.0 - held_drop
        );
    } else if held_trim < held_drop - 20.0 {
        eprintln!(
            "(b) YES — VERIFIED. {held_drop:.1}% of the bytes survived dropping EVERY handle, \
             and malloc_trim(0) then returned them, leaving {held_trim:.1}%. The program owns \
             nothing at that point; glibc was sitting on the pages.\n\
             \n\
             CONSEQUENCE, and it is the reason two eviction fixes changed nothing: eviction \
             and trim are BOTH required. Evicting without trimming frees bytes the OS never \
             gets back, so RSS stays flat and the fix looks like it failed. Trimming without \
             evicting has nothing to trim, because a bound that never fires never frees."
        );
    } else {
        eprintln!(
            "(b) NO: {held_drop:.1}% survived the drop AND {held_trim:.1}% survived \
             malloc_trim. Something still owns these bytes after every handle is gone. \
             Find the owner."
        );
    }
    eprintln!();
}