gunnar-store 1.6.0

//! Measure, never assert, made a test: **the conformance suite is seen to go red.**
//!
//! Every backend passes the suite. That is worth exactly nothing until the suite
//! has been shown to catch a backend that is wrong, so the wrappers below break
//! one in five specific, realistic ways and run the *same* suite over the
//! result. Each test names the checks that must fail and, just as importantly,
//! the checks that must still pass, because a suite that fails everything is as
//! blind as one that passes everything.
//!
//! The five breaks are not arbitrary. Each is a bug someone would plausibly
//! ship, and three of them are bugs that *have* shipped in this tree's history:
//!
//! | break | the plausible bug |
//! |---|---|
//! | `Cas` | `update_ref` that applies without consulting `old` — every push wins |
//! | `UnderSend` | `reachable` that contributes only the tips: the clone exits zero and is short by everything beneath them |
//! | `StoredSize` | `size` that reports the stored extent instead of the post-resolution size — the quota gate that can never fire |
//! | `Truncate` | `get` that returns not quite the bytes it stored |
//! | `Regenerate` | `get_or_create` that always creates: the SSH host key changes on every boot |
//! | `HeadIsARow` | `HEAD` advertised in the ref stream, `head()` answering nothing — **the disagreement the two arms actually had**, and the one the suite could not see until 2026-08-10 |
//! | `PackCloses` | `emit_pack` treating its object set as tips and closing over the graph — Agent K found this live: one commit in, four objects out on a `--filter=tree:0` fetch |
//!
//! **`UnderSend` is the one that matters most.** It passes every object check,
//! every ref check, `gc` and the restart, because nothing else in the suite asks
//! the graph a question. It is also the failure mode with no external symptom —
//! *"the clone ends with a pack the client cannot close, or worse, closes against
//! objects it happened to have"* — so a suite without a genuine closure
//! assertion would call it green and a fetch would call it done.
//!
//! **`PackCloses` is the sharp one's serving twin.** The over-sent pack passes
//! `git index-pack --strict` **and** `git fsck`, the clone succeeds, and the
//! client receives objects it explicitly asked not to have — so no exit code and
//! no format check anywhere in this tree can see it. Only an assertion on what
//! came out can, which is why `emit_pack_emits_the_set_it_was_given` asserts
//! membership rather than validity.
//!
//! **`HeadIsARow` is the one that proves the newest check is live.** Before it,
//! the two `GitOps` arms disagreed about whether `HEAD` is a row for as long as
//! there were two arms, and this suite was **incapable of noticing**, because it
//! never exercised `HEAD` at all. A check added for a defect nobody can
//! reproduce is a check nobody can trust; this reproduces it.
//!
//! **`StoredSize` is the quiet one.** The number it corrupts is read by exactly
//! one caller, an object-size ceiling, and on every corpus in this repository
//! that ceiling never fires — so the bug is invisible to every wire-byte and
//! object-count test and shows up in production as an oversized object served.
//! It takes a fixture with a delta in it to see at all.
//!
//! The wrappers are generic over any [`Harness`], which buys the other half of
//! the argument: breaking a *real* harness and watching the same checks go red
//! proves that green run is live code and not a harness that quietly does
//! nothing. Since 2026-08-10 the only harness in this crate is the control, and
//! the real ones are elsewhere — `gunnar-core`'s Arrow-log harness reconstructs
//! `Regenerate` locally, and both `GitOps` backends should do the same in their
//! own repositories.

use std::collections::BTreeSet;

use git_storage_trait::{
    Caps, Extent, GcReport, GitOps, GitServe, ObjType, Oid, PackStats, ReachSet, RefCas, RefRow,
    RefUpdate, Stored, TxId,
};

use super::{run_all, Backend, Harness, Outcome};
use crate::error::Result;
use crate::secret::SecretStore;

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Break {
    /// Nothing broken. Proves the wrappers themselves are transparent.
    None,
    /// `update_ref` ignores the `old` it was given.
    Cas,
    /// `reachable` contributes the tips and nothing beneath them.
    UnderSend,
    /// `size` and `Stored::uncompressed_size` report the stored entry length.
    StoredSize,
    /// `get` returns one byte less than was stored.
    Truncate,
    /// `get_or_create` always creates.
    Regenerate,
    /// `HEAD` is a row in `refs()` and `head()` cannot answer.
    HeadIsARow,
    /// `emit_pack` closes over the graph instead of emitting the set it was
    /// given.
    PackCloses,
    /// `reachable` answers a non-commit `want` with its own id and stops, so an
    /// annotated tag arrives without the commit it names.
    TagStopsAtItself,
    /// `put_pack_observed` declares that it publishes a live object count and
    /// then publishes `(0, N)` and `(N, N)` — the two endpoints, and nothing
    /// between them.
    TwoValueCount,
}

struct BrokenGit<G> {
    inner: G,
    brk: Break,
}

// `GitServe` and not `GitOps`, because two of the breaks are about the reading
// half and one of them — `HeadIsARow` — has to reach across: the bug it models
// moves `HEAD` from the accessor into the row stream, so the `refs()` override
// asks `head()`.
impl<G: GitServe> GitOps for BrokenGit<G> {
    fn put(&self, pack: &[u8], refs: &[RefUpdate]) -> anyhow::Result<TxId> {
        self.inner.put(pack, refs)
    }

    fn put_pack(&self, bytes: &[u8]) -> anyhow::Result<TxId> {
        self.inner.put_pack(bytes)
    }

    /// **The gap that was shipped, wrapped in a struct.**
    ///
    /// This is not a hypothetical break: it is the literal behaviour every
    /// engine had until 2026-09-08 — `(0, N)` at the packfile header, `(N, N)`
    /// when the ingest returned, nothing in between — dressed in a
    /// `publishes_pack_progress()` that says `true`. A console watching it sees
    /// a bar that jumps from empty to full and learns nothing during the
    /// minutes that matter.
    ///
    /// It reads the object count out of the pack's own header (bytes 8..12)
    /// rather than counting anything, which is the point: **a fabricated series
    /// costs nothing to produce and is indistinguishable from a real one at
    /// every seam except this check.** Twelve bytes is also all it may read —
    /// a break that walked the pack to fake the numbers would be a second pack
    /// decoder, which is the thing the real implementation exists to avoid.
    fn put_pack_observed(
        &self,
        bytes: &[u8],
        progress: &dyn git_storage_trait::PackProgress,
    ) -> anyhow::Result<TxId> {
        if self.brk != Break::TwoValueCount {
            return self.inner.put_pack_observed(bytes, progress);
        }
        let total = u64::from(u32::from_be_bytes([
            bytes[8], bytes[9], bytes[10], bytes[11],
        ]));
        progress.indexed(0, total);
        let tx = self.inner.put_pack(bytes)?;
        progress.indexed(total, total);
        Ok(tx)
    }

    fn publishes_pack_progress(&self) -> bool {
        self.brk == Break::TwoValueCount || self.inner.publishes_pack_progress()
    }

    fn put_refs(&self, updates: &[RefUpdate]) -> anyhow::Result<TxId> {
        self.inner.put_refs(updates)
    }

    fn get(&self, oid: Oid<'_>) -> anyhow::Result<Option<Stored>> {
        let Some(mut stored) = self.inner.get(oid)? else {
            return Ok(None);
        };
        match self.brk {
            Break::Truncate => {
                stored.bytes.pop();
            }
            // The bug: the *stored* length, which for a delta is the length of
            // the instructions rather than of what they resolve to.
            Break::StoredSize => stored.uncompressed_size = stored.bytes.len() as u64,
            _ => {}
        }
        Ok(Some(stored))
    }

    fn has(&self, oid: Oid<'_>) -> anyhow::Result<bool> {
        self.inner.has(oid)
    }

    fn size(&self, oid: Oid<'_>) -> anyhow::Result<Option<u64>> {
        if self.brk == Break::StoredSize {
            return Ok(self.inner.get(oid)?.map(|s| s.bytes.len() as u64));
        }
        self.inner.size(oid)
    }

    fn extents(&self, oids: &[Oid<'_>]) -> anyhow::Result<Vec<Option<Extent>>> {
        self.inner.extents(oids)
    }

    fn refs(&self) -> anyhow::Result<Vec<RefRow>> {
        let mut rows = self.inner.refs()?;
        if self.brk == Break::HeadIsARow {
            // The bug: the pseudo-ref in the row stream. It is the shape the
            // Arrow arm actually had — `HEAD` was written to the ref log like
            // any other row and read back out with the rest of them.
            if let Some(head) = self.inner.head()? {
                rows.push(head);
                rows.sort_by(|a, b| a.name.cmp(&b.name));
            }
        }
        Ok(rows)
    }

    fn update_ref(
        &self,
        name: &str,
        old: Option<Oid<'_>>,
        new: Option<Oid<'_>>,
    ) -> anyhow::Result<TxId> {
        if self.brk == Break::Cas {
            // The bug: `old` is never consulted. Whatever is there is accepted
            // as what was expected, so every push wins.
            let current = self
                .inner
                .refs()?
                .into_iter()
                .find(|r| r.name == name)
                .and_then(|r| r.oid);
            return self.inner.update_ref(name, current.as_deref(), new);
        }
        self.inner.update_ref(name, old, new)
    }

    fn reachable(&self, want: &[Oid<'_>], have: &[Oid<'_>]) -> anyhow::Result<Vec<Vec<u8>>> {
        if self.brk == Break::UnderSend {
            // The bug: a tip whose closure the index does not actually hold,
            // contributing only itself. Deduplicated, because an answer with a
            // repeat in it would be caught by something other than the bug.
            let tips: BTreeSet<Vec<u8>> = want.iter().map(|id| id.to_vec()).collect();
            return Ok(tips.into_iter().collect());
        }
        if self.brk == Break::TagStopsAtItself {
            // **The bug the appliance shipped**, modelled exactly: the store's
            // reachability index is built over the commit graph, so a `want`
            // that is a commit gets its whole closure and a `want` that is not
            // — an annotated tag — contributes its own id and nothing else. A
            // tree never points at a tag, so no other tip picks it up either.
            //
            // Every commit still answers correctly, which is the whole reason
            // this went unnoticed: the four checks around it stay green, six of
            // the reference appliance's 25 repositories could not obtain a second copy,
            // and `git clone` worked throughout.
            let mut commits: Vec<Oid<'_>> = Vec::new();
            let mut out: BTreeSet<Vec<u8>> = BTreeSet::new();
            for id in want {
                match self.inner.header(id)? {
                    Some((ObjType::Commit, _)) => commits.push(id),
                    _ => {
                        out.insert(id.to_vec());
                    }
                }
            }
            out.extend(self.inner.reachable(&commits, have)?);
            return Ok(out.into_iter().collect());
        }
        self.inner.reachable(want, have)
    }

    fn put_refs_cas(&self, edits: &[RefCas<'_>]) -> anyhow::Result<TxId> {
        if self.brk == Break::Cas {
            // The same bug as `update_ref`'s, in the batch shape: the caller's
            // expectation is replaced by whatever is there, so no edit can lose.
            let held = self.inner.refs()?;
            let current: Vec<Option<Vec<u8>>> = edits
                .iter()
                .map(|e| {
                    held.iter()
                        .find(|r| r.name == e.name)
                        .and_then(|r| r.oid.clone())
                })
                .collect();
            let rewritten: Vec<RefCas<'_>> = edits
                .iter()
                .zip(&current)
                .map(|(e, current)| RefCas {
                    name: e.name.clone(),
                    old: current.as_deref(),
                    new: e.new,
                    peeled: e.peeled.clone(),
                })
                .collect();
            return self.inner.put_refs_cas(&rewritten);
        }
        self.inner.put_refs_cas(edits)
    }

    fn gc(&self) -> anyhow::Result<GcReport> {
        self.inner.gc()
    }
}

impl<G: GitServe> GitServe for BrokenGit<G> {
    fn read(&self, oid: Oid<'_>) -> anyhow::Result<Option<(ObjType, Vec<u8>)>> {
        self.inner.read(oid)
    }

    fn header(&self, oid: Oid<'_>) -> anyhow::Result<Option<(ObjType, u64)>> {
        self.inner.header(oid)
    }

    fn sizes(&self, oids: &[Oid<'_>]) -> anyhow::Result<Vec<Option<u64>>> {
        self.inner.sizes(oids)
    }

    fn head(&self) -> anyhow::Result<Option<RefRow>> {
        if self.brk == Break::HeadIsARow {
            // The other half of the same bug, and the half that makes it fatal:
            // a backend that publishes HEAD as a row has no reason to answer
            // the accessor, so the one caller that needs HEAD — the
            // advertisement — gets nothing.
            return Ok(None);
        }
        self.inner.head()
    }

    fn set_head(&self, target: &str) -> anyhow::Result<TxId> {
        self.inner.set_head(target)
    }

    fn emit_pack(
        &self,
        objects: &[Oid<'_>],
        have: &[Oid<'_>],
        caps: &Caps,
        out: &mut dyn std::io::Write,
    ) -> anyhow::Result<PackStats> {
        if self.brk == Break::PackCloses {
            // The bug: treat the exact set as tips and close over it. Every
            // object emitted is reachable, the pack is valid, `fsck` is happy,
            // and the client receives what its filter excluded.
            let closure = self.inner.reachable(objects, have)?;
            let refs: Vec<Oid<'_>> = closure.iter().map(|id| id.as_slice()).collect();
            return self.inner.emit_pack(&refs, have, caps, out);
        }
        self.inner.emit_pack(objects, have, caps, out)
    }

    fn select(&self, want: &[Oid<'_>], have: &[Oid<'_>]) -> anyhow::Result<Option<ReachSet>> {
        self.inner.select(want, have)
    }
}

struct BrokenSecrets<S> {
    inner: S,
    brk: Break,
}

impl<S: SecretStore> SecretStore for BrokenSecrets<S> {
    fn get(&self, key: &str) -> Result<Option<Vec<u8>>> {
        self.inner.get(key)
    }

    fn put(&self, key: &str, value: &[u8]) -> Result<()> {
        self.inner.put(key, value)
    }

    fn delete(&self, key: &str) -> Result<()> {
        self.inner.delete(key)
    }

    fn get_or_create(&self, key: &str, mk: &mut dyn FnMut() -> Result<Vec<u8>>) -> Result<Vec<u8>> {
        if self.brk == Break::Regenerate {
            // The bug: a fresh host key on every call, and therefore on every
            // boot. This is the exact failure the secret-store seam exists to prevent.
            let value = mk()?;
            self.inner.put(key, &value)?;
            return Ok(value);
        }
        self.inner.get_or_create(key, mk)
    }
}

/// Any harness, with one thing about it broken.
struct BrokenHarness<H> {
    inner: H,
    brk: Break,
}

impl<H: Harness> Harness for BrokenHarness<H> {
    type Git = BrokenGit<H::Git>;
    type Secrets = BrokenSecrets<H::Secrets>;

    fn name(&self) -> &str {
        self.inner.name()
    }

    fn open(
        &self,
        slot: &str,
        hash_kind: gix_hash::Kind,
    ) -> anyhow::Result<Backend<Self::Git, Self::Secrets>> {
        let backend = self.inner.open(slot, hash_kind)?;
        Ok(Backend {
            git: BrokenGit {
                inner: backend.git,
                brk: self.brk,
            },
            secrets: BrokenSecrets {
                inner: backend.secrets,
                brk: self.brk,
            },
        })
    }
}

fn red_checks<H: Harness>(inner: H, brk: Break) -> (BTreeSet<&'static str>, Vec<Outcome>) {
    let outcomes = run_all(&BrokenHarness { inner, brk });
    let failed = outcomes
        .iter()
        .filter(|o| !o.passed())
        .map(|o| o.check)
        .collect();
    (failed, outcomes)
}

#[track_caller]
fn assert_red(
    backend: &str,
    failed: &BTreeSet<&'static str>,
    must_fail: &[&str],
    must_pass: &[&str],
) {
    for check in must_fail {
        assert!(
            failed.contains(check),
            "[{backend}] the conformance suite did NOT catch a deliberately broken backend: \
             `{check}` passed when it had to fail. Failing checks were {failed:?}"
        );
    }
    for check in must_pass {
        assert!(
            !failed.contains(check),
            "[{backend}] `{check}` failed against a backend that is not broken in that way. \
             the suite is failing indiscriminately, which is its own kind of blindness. \
             Failing checks were {failed:?}"
        );
    }
}

const CAS_MUST_FAIL: &[&str] = &[
    "update_ref_creates_only_when_absent",
    "update_ref_is_a_real_compare_and_swap",
    "update_ref_delete_checks_its_old_value",
    "put_refs_cas_applies_every_edit_or_none",
];
const CAS_MUST_PASS: &[&str] = &[
    "pack_roundtrip_every_entry",
    "refs_round_trip",
    "reachable_returns_the_whole_closure",
    "secret_get_or_create_survives_restart",
];

#[test]
fn the_wrappers_themselves_are_transparent() {
    // Without this, a red result below could be an artefact of the wrapper
    // rather than of the break it is supposed to model.
    let (failed, outcomes) = red_checks(super::MemHarness::default(), Break::None);
    assert!(
        failed.is_empty(),
        "the pass-through broken-backend wrapper does not itself conform: {failed:?}"
    );
    assert!(!outcomes.is_empty(), "the suite ran nothing");
}

#[test]
fn the_suite_catches_a_broken_compare_and_swap() {
    let (failed, _) = red_checks(super::MemHarness::default(), Break::Cas);
    assert_red("mem", &failed, CAS_MUST_FAIL, CAS_MUST_PASS);
}

/// **The sharp one.** Every object check, every ref check, `gc` and the restart
/// pass: nothing but the closure assertions can see this bug, and a clone served
/// by it exits zero.
#[test]
fn the_suite_catches_a_reachability_index_that_under_sends() {
    let (failed, _) = red_checks(super::MemHarness::default(), Break::UnderSend);
    assert_red(
        "mem",
        &failed,
        &[
            "reachable_returns_the_whole_closure",
            "reachable_subtracts_what_the_client_has",
            "reachable_tolerates_a_have_it_does_not_know",
            "reachable_does_not_follow_a_submodule",
        ],
        &[
            "pack_roundtrip_every_entry",
            "get_returns_the_verbatim_entry_bytes",
            "update_ref_is_a_real_compare_and_swap",
            "gc_keeps_everything_a_ref_points_at",
            "the_store_survives_a_restart",
            "secret_put_get_delete",
        ],
    );
}

/// **The one that was shipping.** `reachable` is right about every commit and
/// wrong about exactly one edge, and the other three closure checks cannot see
/// it because none of them puts a tag in a `want`.
///
/// 🔴 This is the reference appliance on 2026-08-31, in the suite: six of 25
/// repositories refused replication with `missing necessary objects: <commit>
/// is not in this repository (peeling a tag)`, while clones of the same
/// repositories succeeded. `reachable_follows_an_annotated_tag_to_its_target`
/// is the check that had to exist for a suite to call that backend broken, and
/// this is the proof it can.
#[test]
fn the_suite_catches_a_reachability_that_stops_at_a_tag() {
    let (failed, _) = red_checks(super::MemHarness::default(), Break::TagStopsAtItself);
    assert_red(
        "mem",
        &failed,
        &["reachable_follows_an_annotated_tag_to_its_target"],
        &[
            // The three that were green on the appliance the whole time it was
            // failing. If any of these goes red the new check is not what
            // caught the break, and the suite is failing indiscriminately.
            "reachable_returns_the_whole_closure",
            "reachable_subtracts_what_the_client_has",
            "reachable_does_not_follow_a_submodule",
            "pack_roundtrip_every_entry",
            "update_ref_is_a_real_compare_and_swap",
            "gc_keeps_everything_a_ref_points_at",
            "the_store_survives_a_restart",
        ],
    );
}

/// **The quiet one.** Only a fixture with a delta in it can see this, which is
/// why the fixture has one.
#[test]
fn the_suite_catches_a_size_that_is_not_post_resolution() {
    let (failed, _) = red_checks(super::MemHarness::default(), Break::StoredSize);
    assert_red(
        "mem",
        &failed,
        &["size_is_post_resolution_not_stored_length"],
        &[
            "get_returns_the_verbatim_entry_bytes",
            "update_ref_is_a_real_compare_and_swap",
            "reachable_returns_the_whole_closure",
            "secret_put_get_delete",
        ],
    );
}

#[test]
fn the_suite_catches_a_store_that_loses_bytes() {
    let (failed, _) = red_checks(super::MemHarness::default(), Break::Truncate);
    assert_red(
        "mem",
        &failed,
        &["get_returns_the_verbatim_entry_bytes"],
        &[
            "update_ref_is_a_real_compare_and_swap",
            "reachable_returns_the_whole_closure",
            "secret_put_get_delete",
        ],
    );
}

/// **The check that did not exist until 2026-08-10, seen red.**
///
/// `HEAD` in the row stream and `head()` answering nothing is not a
/// hypothetical: it is what one of the two arms did, and the reason it survived
/// is that the suite never asked. This is the proof that asking works.
#[test]
fn the_suite_catches_a_backend_that_calls_head_a_row() {
    let (failed, _) = red_checks(super::MemHarness::default(), Break::HeadIsARow);
    assert_red(
        "mem",
        &failed,
        &["head_is_not_a_row_and_has_its_own_accessor"],
        &[
            "refs_round_trip",
            "update_ref_is_a_real_compare_and_swap",
            "pack_roundtrip_every_entry",
            "reachable_returns_the_whole_closure",
            "emit_pack_emits_the_set_it_was_given",
        ],
    );
}

/// **An `emit_pack` that closes over the graph, seen red.**
///
/// The pack it produces is valid in every mechanical sense — `index-pack
/// --strict` and `fsck` both pass it — so `pack_roundtrip_every_entry` and the
/// closure checks stay green and only the membership assertion fails. That is
/// the point: this bug has no other detector.
#[test]
fn the_suite_catches_an_emit_pack_that_closes_over_its_input() {
    let (failed, _) = red_checks(super::MemHarness::default(), Break::PackCloses);
    assert_red(
        "mem",
        &failed,
        &["emit_pack_emits_the_set_it_was_given"],
        &[
            "read_is_inflated_and_delta_resolved",
            "header_and_sizes_agree_with_read",
            "select_agrees_with_the_graph_or_declines",
            "reachable_returns_the_whole_closure",
            "pack_roundtrip_every_entry",
            "head_is_not_a_row_and_has_its_own_accessor",
        ],
    );
}

/// **The gap of 2026-09-08, seen red.**
///
/// 🔴 `mirror_queue`'s module doc said it in as many words: *"the progress pair
/// goes `(0, N) → (N, N)`"*. A mirror of a large repository published its
/// denominator, then nothing, then its numerator — and every other check in
/// this suite was green throughout, because storing a pack correctly and
/// **saying how far you have got while you store it** are different claims.
///
/// This break declares the second claim and delivers the first, which is the
/// only shape of this bug anybody would actually ship: nobody writes a progress
/// sink that lies about the numbers, they write one that is technically correct
/// at both ends and useless in between.
#[test]
fn the_suite_catches_a_count_that_only_has_two_values() {
    let (failed, _) = red_checks(super::MemHarness::default(), Break::TwoValueCount);
    assert_red(
        "mem",
        &failed,
        &["put_pack_observed_publishes_a_count_that_moves"],
        &[
            // The pack still lands, and every reading is inside the contract:
            // monotone, right denominator, ending at N. Only "it moved" fails,
            // which is the whole point — nothing else in this suite can see it.
            "put_pack_is_idempotent",
            "pack_roundtrip_every_entry",
            "get_returns_the_verbatim_entry_bytes",
            "update_ref_is_a_real_compare_and_swap",
            "reachable_returns_the_whole_closure",
            "the_store_survives_a_restart",
        ],
    );
}

#[test]
fn the_suite_catches_a_regenerated_host_key() {
    let (failed, _) = red_checks(super::MemHarness::default(), Break::Regenerate);
    assert_red(
        "mem",
        &failed,
        &[
            "secret_get_or_create_survives_restart",
            "secret_get_or_create_is_atomic",
        ],
        &[
            "secret_put_get_delete",
            "update_ref_is_a_real_compare_and_swap",
            "pack_roundtrip_every_entry",
        ],
    );
}

// The two Arrow-log red tests moved to `gunnar-core` with the harnesses they
// break (`crates/gunnar-core/tests/arrow_log_conformance.rs`). They asserted
// that the Arrow log's green run is live code rather than a harness that
// quietly does nothing — which matters more there than it would for a database,
// because the log's atomicity comes from the IPC framing rather than from a
// commit, so "the frame makes it atomic" is exactly the claim that goes
// untested. That argument survived the move; `Break::Regenerate` is
// reconstructed there in twenty lines because the decorators here are private
// to this crate and widening them is a decision about the conformance suite's
// public surface, not about where the Arrow log lives.