gunnar-server 4.0.0

//! **The gix arm: `storage-git-gix`, `gix-odb` / `gix-ref` on a plain filesystem.**
//!
//! The second implementation of the same eleven. Not the default and not in the
//! appliance image: `--no-default-features --features server,engine-gix` builds
//! it alone, `--features engine-znippy,engine-gix` builds both and picks at
//! runtime. Its reason for existing is that an A/B needs two things to compare,
//! and until it landed there was one.
//!
//! The store is an ordinary **bare git repository**, so whatever this arm writes,
//! stock `git` can read — which is what makes it a usable oracle for the other.
//!
//! # Three narrowings, recorded as facts rather than as trait-change requests
//!
//! Two are the backend's own, from the two-engine record:
//!
//! 1. `extents()` returns a **pack-local** offset. With N packs an `Extent` does
//!    not identify a byte range on its own, and there is nowhere in `(u64, u64)`
//!    for a pack id. No synthetic global offset was invented, because it would
//!    index nothing.
//! 2. `TxId.push_seq` is **in-process only**. This backend keeps no ref log, so
//!    there is no durable ordering authority; it resets on restart.
//!
//! The third belongs to this file and is new with it:
//!
//! 3. **`want_objects` is refused.** Enumerating the ids one push introduced is
//!    not on `GitOps` and has no analog here — the znippy arm answers it out of
//!    its own extent index (`oids_in_extent`), and this arm has no extent index
//!    to ask. [`ingest`] therefore *fails* rather than answering `None`: `None`
//!    on that field means **not asked for**, and a policy gate handed `None`
//!    refuses, so returning it here would turn "this engine cannot tell you"
//!    into "your policy passed". A named, bounded absence, never a plausible
//!    wrong answer.
//!
//! # `seal` has no analog either, and that is not a gap
//!
//! `seal()` folds znippy's live logs into reserved **Arrow** sections. This
//! backend has no live logs to fold — `gc()` reads the packs and the ref store
//! directly — so [`maintain`] reports `sections_sealed: 0`, which is the count
//! of what happened rather than a stub value.

use std::io::BufRead;
use std::path::{Path, PathBuf};
use std::sync::Arc;

use anyhow::{bail, Context, Result};
use git_storage_trait::{GitOps, GitServe, Oid, TxId};
use storage_git_gix::GixStore;

use super::store_cache::{self, Cache};
use super::{BeforeStore, MaintenanceReport, OpenRepo, PushReceipt};

/// Every `GixStore` this process has opened, by store root.
static STORES: Cache<GixStore> = Cache::new();

/// **The one handle for `git_dir`** — the reading path and the ingest path get
/// the same object, so a `put_pack` refreshes the view the server serves from.
///
/// This is the fix for the defect written up in [`super::store_cache`]: `open`
/// and `ingest` used to call `GixStore::open_or_init` separately, and
/// `GixStore::reload_packs` — *"git's `reprepare_packed_git()`, called from the
/// writer, which is where git calls it"* — then refreshed a handle that was
/// about to be dropped. Everything pushed into a repository was invisible to
/// the reader for the life of the process.
///
/// It is by construction rather than by a guard (one implementation): there is no
/// "invalidate after write" step to forget, because there is nothing to
/// invalidate. One writer, one reader, one `PackSet`.
///
/// # This says nothing about two PROCESSES
///
/// `storage-git-gix`'s own docs record *"no writer lock — two processes writing
/// one store root is not defended against here"*, and that is still true and
/// still not defended against. This cache is per-process and fixes a
/// per-process defect. The file's original note — that one handle per open was
/// "the honest shape of a store that makes no such promise" — conflated the two:
/// declining an in-process cache never bought any cross-process safety, and it
/// cost correctness within the process.
fn open_shared(git_dir: &Path, hash_kind: gix_hash::Kind) -> Result<Arc<GixStore>> {
    let root = PathBuf::from(git_dir);
    store_cache::shared(&STORES, root, || {
        GixStore::open_or_init(git_dir, hash_kind)
            .with_context(|| format!("opening the gix store at {}", git_dir.display()))
    })
}

/// Release every cached handle. See [`super::shutdown`].
pub fn shutdown() {
    store_cache::clear(&STORES);
}

/// Open — creating if absent — the bare repository at `git_dir`.
///
/// Through [`open_shared`], so the handle this hands the serving path is the
/// same one [`ingest`] writes through. `ServerStores` also caches the
/// `Arc<OpenRepo>` per repository, but that cache is keyed on the repository
/// and is not what makes the write visible — the shared `GixStore` underneath
/// is.
///
/// # 🔴 This does not compile until edda implements `GitServe` for `GixStore`
///
/// [`OpenRepo::new`] requires the **reading** contract, because everything the
/// server does with a repository needs it (`UploadPack`, `ReceivePack`,
/// `gunnar-policy`'s ancestry walk, and `HEAD`, which is not a row in
/// [`GitOps::refs`]). `storage-git-gix` implements `GitOps` and not `GitServe`
/// — the arm is PAUSED — so this call is a compile
/// error in any build carrying `engine-gix`, and it is the right kind: a
/// missing half of the contract, named at the seam, rather than a repository
/// that opens and then refuses every clone at runtime.
///
/// Two things are owed by edda before this arm builds, and neither is gunnar's
/// (see `agentL-report.md`):
///
/// 1. `GitOps::put_refs_cas` — a **pre-existing** `E0046` on `GixStore`, which
///    is why `--features engine-gix` did not compile before this change either.
/// 2. `impl GitServe for GixStore`.
pub fn open(git_dir: &Path, hash_kind: gix_hash::Kind) -> Result<Arc<OpenRepo>> {
    Ok(OpenRepo::new(
        git_dir,
        hash_kind,
        Arc::new(Served(open_shared(git_dir, hash_kind)?)),
    ))
}

/// **The gix store as the server serves it**: every call goes straight
/// through, except one `select` the engine should have declined (lane
/// GUN-PERF-FETCH, 2026-10-31).
///
/// `GitServe::select`'s contract says `Ok(None)` means *"this engine cannot
/// answer cheaply — walk it yourself"*. Without a reachability bitmap,
/// `storage-git-gix` 0.3.0 answers a request with `have`s anyway, by closing
/// over the client's whole `have` side — every object the client holds.
/// MEASURED on oden, git.git (81 188 commits, 1 006 refs), one new commit
/// fetched: **23 s** of server CPU, where gunnar's own walk (git's `limit_list`
/// plus the edge's trees) serves the same request in **0.2 to 0.4 s**.
///
/// So such a request is declined here and the walk answers it. Two shapes stay
/// with the engine: a request with no `have` (a clone, where both cost the same
/// whole walk), and every request once [`GitOps::gc`] has written the one
/// `.bitmap` the engine reads (its bitmap cache uses a
/// pack directory only when it holds exactly one). The fix belongs in the
/// engine; this is the server keeping the contract on its behalf until it does.
struct Served(Arc<GixStore>);

impl Served {
    /// The engine's own rule for "a bitmap is there to read": exactly one
    /// `.bitmap` in `objects/pack`.
    fn has_bitmap(&self) -> bool {
        let Ok(entries) = std::fs::read_dir(self.0.git_dir().join("objects").join("pack")) else {
            return false;
        };
        entries
            .flatten()
            .filter(|entry| entry.path().extension().is_some_and(|e| e == "bitmap"))
            .count()
            == 1
    }
}

impl GitOps for Served {
    fn put(&self, pack: &[u8], refs: &[git_storage_trait::RefUpdate]) -> Result<TxId> {
        self.0.put(pack, refs)
    }
    fn put_pack(&self, bytes: &[u8]) -> Result<TxId> {
        self.0.put_pack(bytes)
    }
    fn put_pack_observed(
        &self,
        bytes: &[u8],
        progress: &dyn git_storage_trait::PackProgress,
    ) -> Result<TxId> {
        self.0.put_pack_observed(bytes, progress)
    }
    fn publishes_pack_progress(&self) -> bool {
        self.0.publishes_pack_progress()
    }
    fn put_refs(&self, updates: &[git_storage_trait::RefUpdate]) -> Result<TxId> {
        self.0.put_refs(updates)
    }
    fn get(&self, oid: Oid<'_>) -> Result<Option<git_storage_trait::Stored>> {
        self.0.get(oid)
    }
    fn has(&self, oid: Oid<'_>) -> Result<bool> {
        self.0.has(oid)
    }
    fn size(&self, oid: Oid<'_>) -> Result<Option<u64>> {
        self.0.size(oid)
    }
    fn extents(&self, oids: &[Oid<'_>]) -> Result<Vec<Option<git_storage_trait::Extent>>> {
        self.0.extents(oids)
    }
    fn refs(&self) -> Result<Vec<git_storage_trait::RefRow>> {
        self.0.refs()
    }
    fn update_ref(&self, name: &str, old: Option<Oid<'_>>, new: Option<Oid<'_>>) -> Result<TxId> {
        self.0.update_ref(name, old, new)
    }
    fn put_refs_cas(&self, edits: &[git_storage_trait::RefCas<'_>]) -> Result<TxId> {
        self.0.put_refs_cas(edits)
    }
    fn reachable(&self, want: &[Oid<'_>], have: &[Oid<'_>]) -> Result<Vec<Vec<u8>>> {
        self.0.reachable(want, have)
    }
    fn gc(&self) -> Result<git_storage_trait::GcReport> {
        self.0.gc()
    }
}

impl GitServe for Served {
    fn read(&self, oid: Oid<'_>) -> Result<Option<(git_storage_trait::ObjType, Vec<u8>)>> {
        self.0.read(oid)
    }
    fn header(&self, oid: Oid<'_>) -> Result<Option<(git_storage_trait::ObjType, u64)>> {
        self.0.header(oid)
    }
    fn sizes(&self, oids: &[Oid<'_>]) -> Result<Vec<Option<u64>>> {
        self.0.sizes(oids)
    }
    fn index_now(&self) -> Result<usize> {
        self.0.index_now()
    }
    fn head(&self) -> Result<Option<git_storage_trait::RefRow>> {
        self.0.head()
    }
    fn set_head(&self, target: &str) -> Result<TxId> {
        self.0.set_head(target)
    }
    fn set_head_projected(&self, target: &str) -> Result<TxId> {
        self.0.set_head_projected(target)
    }
    fn emit_pack(
        &self,
        objects: &[Oid<'_>],
        have: &[Oid<'_>],
        caps: &git_storage_trait::Caps,
        out: &mut dyn std::io::Write,
    ) -> Result<git_storage_trait::PackStats> {
        self.0.emit_pack(objects, have, caps, out)
    }
    /// See [`Served`]: a request with `have`s on a store without a bitmap is
    /// the walk's.
    fn select(
        &self,
        want: &[Oid<'_>],
        have: &[Oid<'_>],
    ) -> Result<Option<git_storage_trait::ReachSet>> {
        if !have.is_empty() && !self.has_bitmap() {
            return Ok(None);
        }
        self.0.select(want, have)
    }
}

/// One push into the bare repository: `index_pack` writes `.pack` / `.idx`.
///
/// See narrowing 3 in the module docs for `want_objects`.
///
/// `progress` is carried and handed to
/// [`GitOps::put_pack_observed`](git_storage_trait::GitOps::put_pack_observed),
/// which on this arm is the trait's own default: `GixStore` has no entry loop
/// of its own — `gix_pack` owns the split — so it publishes nothing and says so
/// through `publishes_pack_progress()`. That is the right answer rather than a
/// gap: a fabricated `(0, N) → (N, N)` here would read to a console exactly
/// like the live count the znippy arm now produces.
pub fn ingest(
    git_dir: &Path,
    hash_kind: gix_hash::Kind,
    pack: &mut dyn BufRead,
    objects: super::ObjectSet,
    progress: Option<&dyn git_storage_trait::PackProgress>,
    before_store: Option<BeforeStore<'_>>,
) -> Result<PushReceipt> {
    // **`Required` is refused; `Wanted` is answered with `None`.**
    //
    // It used to refuse on a bare `want_objects`, and when the push attestation
    // started asking for the set that turned every push on this arm into
    // "the repository is unavailable". The refusal is still right for a gate —
    // a gate over objects cannot decide without them — and wrong for a record
    // that has a field for saying the set is unknown.
    if objects == super::ObjectSet::Required {
        bail!(
            "the gix storage engine cannot enumerate the objects a push introduced: that is not \
             one of the eleven, and this backend keeps no extent index to answer it from. The \
             policy on this repository has a gate over the pushed object set, so the push is \
             refused rather than passed on an answer the engine did not give. Serve this \
             repository from the znippy engine, or drop the gate."
        );
    }
    let started = std::time::Instant::now();
    // THE SAME handle the serving path holds. `put_pack` reloads the pack set
    // on the instance it is called on, so writing through a throwaway here is
    // what made every pushed object invisible until the next restart.
    let store = open_shared(git_dir, hash_kind)?;
    let pushed = super::read_pushed_pack_spooling(pack, hash_kind, Some(git_dir))?;
    // The transaction log first (`Engine::ingest_push`'s `before_store`).
    if let Some(hook) = before_store {
        hook(&pushed.bytes).context("framing the pushed pack in the transaction log")?;
    }
    match progress {
        Some(sink) => store.put_pack_observed(&pushed.bytes, sink),
        None => store.put_pack(&pushed.bytes),
    }
    .context("storing the pushed pack")?;
    let pack_id = super::pack_trailer(&pushed.bytes, hash_kind)?;
    Ok(PushReceipt {
        num_objects: pushed.num_objects,
        version: pushed.version,
        pack_id,
        pack_bytes: pushed.bytes.len() as u64,
        new_oids: None,
        elapsed_ms: started.elapsed().as_millis() as u64,
    })
}

/// `gc` alone. There is no seal on this arm — see the module docs.
pub fn maintain(git_dir: &Path, hash_kind: gix_hash::Kind) -> Result<MaintenanceReport> {
    // Shared for the same reason ingest is, and it is the same bug in the other
    // direction: `gc` UNLINKS packs, so a reader holding its own `PackSet`
    // would go on answering out of an index naming files that are gone.
    let store = open_shared(git_dir, hash_kind)?;
    let report = store.gc().context("sweeping dead packs")?;
    Ok(MaintenanceReport {
        strategy: report.strategy.to_owned(),
        bytes_before: report.bytes_before,
        bytes_after: report.bytes_after,
        rows: report.rows,
        retired_packs: report.retired_packs,
        verified: report.verified,
        sections_sealed: 0,
    })
}

#[cfg(test)]
mod tests {
    use std::process::Command;

    use super::*;

    /// A one-commit pack and the oid of its commit, built by stock `git`.
    ///
    /// Stock git rather than a hand-rolled writer: the bytes this arm has to
    /// accept are the bytes a real client pushes, and a pack this file wrote
    /// itself would prove that this file agrees with itself.
    fn tiny_pack(work: &Path) -> (Vec<u8>, String) {
        let run = |args: &[&str]| {
            let out = Command::new("git")
                .args(args)
                .current_dir(work)
                .output()
                .expect("git is on PATH (xtask preflight asserts it)");
            assert!(
                out.status.success(),
                "git {args:?}: {:?}",
                out.status.code()
            );
            out.stdout
        };
        run(&["init", "-q", "--initial-branch=main", "."]);
        run(&["config", "user.email", "t@t"]);
        run(&["config", "user.name", "t"]);
        std::fs::write(work.join("f"), b"hello").expect("fixture file");
        run(&["add", "f"]);
        run(&["commit", "-qm", "one"]);
        let oid = String::from_utf8(run(&["rev-parse", "HEAD"]))
            .expect("utf8")
            .trim()
            .to_owned();

        // `--revs --stdout` takes the rev on stdin and writes the pack to
        // stdout, which is the same shape `receive-pack` hands `ingest`.
        let mut child = Command::new("git")
            .args(["pack-objects", "--revs", "--stdout", "-q"])
            .current_dir(work)
            .stdin(std::process::Stdio::piped())
            .stdout(std::process::Stdio::piped())
            .spawn()
            .expect("git pack-objects");
        {
            use std::io::Write as _;
            let mut stdin = child.stdin.take().expect("stdin");
            writeln!(stdin, "{oid}").expect("write rev");
        }
        let out = child.wait_with_output().expect("pack-objects output");
        assert!(out.status.success(), "git pack-objects failed");
        assert!(!out.stdout.is_empty(), "git pack-objects produced no pack");
        (out.stdout, oid)
    }

    /// **A push must be visible to a reader that was opened BEFORE it.**
    ///
    /// The regression test for the defect in [`super::super::store_cache`], and
    /// the ordering is the whole test. `open` *after* `ingest` passes even on
    /// the broken code, because a fresh handle loads a fresh `PackSet` off disk
    /// — which is exactly why restarting the server "fixed" it and why this went
    /// unnoticed. The server does not work that way: `ServerStores` caches an
    /// `Arc<OpenRepo>` per repository, so the reader is almost always older than
    /// the push.
    ///
    /// Seen red on the pre-fix code, which answered
    /// `<oid> is a LOOSE object: it is in this repository but has no pack entry,
    /// so there are no stored bytes to copy` — with the object sitting in a pack
    /// that `git verify-pack` called `ok`.
    #[test]
    fn a_push_is_visible_to_a_reader_opened_before_it() {
        let dir = tempfile::tempdir().expect("tempdir");
        let work = dir.path().join("work");
        std::fs::create_dir_all(&work).expect("work dir");
        let (pack, oid_hex) = tiny_pack(&work);

        let git_dir = dir.path().join("repo.git");
        let kind = gix_hash::Kind::Sha1;

        // THE READER FIRST. This is the ordering the server has.
        let reader = open(&git_dir, kind).expect("opening the repository");
        let raw = gix_hash::ObjectId::from_hex(oid_hex.as_bytes()).expect("oid");
        assert!(
            !reader.serve().has(raw.as_slice()).expect("has() before"),
            "the repository already held the commit before it was pushed, so this test would \
             pass without the push and prove nothing"
        );

        let mut cursor: &[u8] = &pack;
        ingest(
            &git_dir,
            kind,
            &mut cursor,
            crate::storage::ObjectSet::NotNeeded,
            // `None`: this test is about the store handle a reader holds, not
            // about progress reporting. RED before this argument was added
            // here — `ingest` grew a fifth parameter and this call did not
            // follow it, so the whole `gix_engine` unit-test target stopped
            // compiling and took the crate's `--lib` tests with it.
            None,
            // `None`: no `before_store` tee either. `ingest` grew a sixth
            // parameter (786bc63, the twin frames a pushed pack) and this call
            // did not follow it, which again took the crate's `--lib` tests down.
            None,
        )
        .expect("ingesting the pushed pack");

        assert!(
            reader.serve().has(raw.as_slice()).expect("has() after"),
            "the reader opened before the push cannot see it: the ingest wrote through a \
             different store handle and refreshed that one instead"
        );
        let got = reader
            .serve()
            .get(raw.as_slice())
            .expect("get() must not error")
            .expect("the commit must have stored bytes to copy");
        assert!(
            !got.bytes.is_empty(),
            "the commit resolved to zero stored bytes, which is what `emit_pack` copies"
        );

        shutdown();
    }

    /// Two commits, the second on the first, as one stock-git pack.
    fn two_commits(work: &Path) -> (Vec<u8>, String, String) {
        let run = |args: &[&str]| {
            let out = Command::new("git")
                .args(args)
                .current_dir(work)
                .output()
                .expect("git is on PATH (xtask preflight asserts it)");
            assert!(
                out.status.success(),
                "git {args:?}: {:?}",
                out.status.code()
            );
            String::from_utf8(out.stdout)
                .expect("utf8")
                .trim()
                .to_owned()
        };
        run(&["init", "-q", "--initial-branch=main", "."]);
        run(&["config", "user.email", "t@t"]);
        run(&["config", "user.name", "t"]);
        std::fs::write(work.join("f"), b"one").expect("fixture file");
        run(&["add", "f"]);
        run(&["commit", "-qm", "one"]);
        let first = run(&["rev-parse", "HEAD"]);
        std::fs::write(work.join("f"), b"two").expect("fixture file");
        run(&["commit", "-qam", "two"]);
        let second = run(&["rev-parse", "HEAD"]);
        let mut child = Command::new("git")
            .args(["pack-objects", "--all", "--revs", "--stdout", "-q"])
            .current_dir(work)
            .stdin(std::process::Stdio::null())
            .stdout(std::process::Stdio::piped())
            .spawn()
            .expect("git pack-objects");
        let pack = std::io::Read::bytes(child.stdout.take().expect("stdout"))
            .collect::<std::io::Result<Vec<u8>>>()
            .expect("read the pack");
        assert!(child.wait().expect("pack-objects").success());
        (pack, first, second)
    }

    /// **An incremental fetch is the walk's to answer until the store has a
    /// reachability bitmap** (lane GUN-PERF-FETCH, 2026-10-31).
    ///
    /// `GitServe::select`'s contract: `Ok(None)` means *"this engine cannot
    /// answer cheaply — walk it yourself"*. `storage-git-gix` 0.3.0 answers
    /// anyway when it has no `.bitmap`, with a walk that closes over the
    /// client's whole `have` side — every object it holds. MEASURED on oden,
    /// git.git (81 188 commits, 1 006 refs), one new commit fetched: **23 s**,
    /// where gunnar's own walk (git's `limit_list` plus the edge) serves the
    /// same request in **0.2 to 0.4 s**. So the server's handle declines for
    /// it: a request with `have`s, on a store with no bitmap, goes to the walk.
    /// A clone (no `have`) costs the same either way and stays with the
    /// engine, and so does every request once `gc` has written a bitmap.
    #[test]
    fn a_fetch_with_haves_on_a_store_without_a_bitmap_is_left_to_the_walk() {
        let dir = tempfile::tempdir().expect("tempdir");
        let work = dir.path().join("work");
        std::fs::create_dir_all(&work).expect("work dir");
        let (pack, first, second) = two_commits(&work);
        let git_dir = dir.path().join("repo.git");
        let kind = gix_hash::Kind::Sha1;
        let reader = open(&git_dir, kind).expect("opening the repository");
        ingest(
            &git_dir,
            kind,
            &mut pack.as_slice(),
            crate::storage::ObjectSet::NotNeeded,
            None,
            None,
        )
        .expect("ingesting the pack");
        let raw = |hex: &str| {
            gix_hash::ObjectId::from_hex(hex.as_bytes())
                .expect("oid")
                .as_slice()
                .to_vec()
        };
        let (first, second) = (raw(&first), raw(&second));

        assert!(
            reader
                .serve()
                .select(&[&second], &[&first])
                .expect("select")
                .is_none(),
            "no bitmap and a `have`: the engine's answer is a closure over everything the client \
             holds, so the server must walk instead"
        );
        assert!(
            reader
                .serve()
                .select(&[&second], &[])
                .expect("select")
                .is_some(),
            "a clone has no `have` side to close over; the engine answers it as before"
        );

        let bitmap = open_shared(&git_dir, kind)
            .expect("the shared handle")
            .write_reachability_bitmap()
            .expect("write a bitmap");
        assert!(
            bitmap.is_some(),
            "one pack, closed: a bitmap must be written"
        );
        assert!(
            reader
                .serve()
                .select(&[&second], &[&first])
                .expect("select")
                .is_some(),
            "with a bitmap the engine answers a fetch cheaply, and it is asked"
        );
        shutdown();
    }
}