znippy_common/

precompressed.rs

//! Deciding whether a payload is **already compressed**, and so should be stored
//! raw instead of handed to the codec.
//!
//! ## Where the probe lives
//!
//! In [`ldeflate::precompressed`], not here. This module is a **re-export** plus
//! the one leg that cannot live there: the path/extension fallback.
//!
//! The direction is forced, not chosen. `znippy-common` already depends on
//! `znippy-zoomies` (`Cargo.toml:96`, plus `ljar`/`lbzip2`/`lgz`), so an
//! `ldeflate -> znippy-common` edge would close a cycle; and `znippy-common`
//! drags `arrow`, `blake3` and an `openzl-sys` whose `build.rs` fetches a tarball
//! over the network — for 88 lines of `std`-only byte matching. The dependency
//! arrow already pointed at zoomies, which makes `ldeflate` the home: it is the
//! leaf, and it is the *compressor*, the only thing that ever needs to ask the
//! question.
//!
//! What is left here is genuinely znippy's: [`SkipPolicy::skip_by_path`] consults
//! [`crate::index::is_probably_compressed`], an extension table that `ldeflate`
//! has no business knowing about — it is handed `&[u8]`, never a `Path`.
//! [`SkipPolicy`] is therefore a thin wrapper that adds the path leg and
//! **delegates** the byte leg to `ldeflate::precompressed::SkipPolicy::skip`.
//! There is one magic table, one `looks_compressed`, one `is_zlib_stream`, and
//! one hint-to-verdict mapping for bytes — in `ldeflate`.
//!
//! ## The resolution order
//!
//! | # | mechanism | cost | how it can be wrong |
//! |---|---|---|---|
//! | 1 | an explicit caller [`ContentHint`] | free | only if the caller lies |
//! | 2 | the extension table ([`crate::index::is_probably_compressed`]) | one `str` compare | the name can lie |
//! | 3 | a [magic-byte probe](looks_compressed) over the first [`SNIFF_PREFIX_LEN`] bytes | ≤ 16 byte compares | a container whose magic is not at offset 0 |
//! | 4 | default | — | — |
//!
//! **The hint is the primary mechanism, not the fallback.** A caller sealing a
//! git packfile *knows* what it is holding; no amount of inspection beats being
//! told. It is deterministic by construction, costs nothing, and carries
//! knowledge the bytes themselves do not.
//!
//! ## Why there is no entropy estimate
//!
//! An entropy estimate was considered and **rejected**. It is a heuristic with a
//! tunable threshold, it costs a scan proportional to the data, and it misjudges
//! in both directions — high-entropy-but-compressible (a table of hashes with a
//! compressible envelope) and low-entropy-but-already-compressed (a short zstd
//! frame) both exist. A wrong guess wastes CPU or wastes bytes and reports
//! neither. Every probe in `ldeflate` is exact: a fixed byte pattern at a fixed
//! offset, no threshold, no sample size to tune.
//!
//! ## What a probe does *not* claim
//!
//! [`looks_compressed`] reads only the head of a buffer. For a file sliced into
//! chunks, only chunk 0 carries the magic. The big-file pass in `znippy-compress`
//! therefore takes chunk 0's verdict and carries it over the whole file — but
//! **excluding** [`is_zlib_stream`], which is a two-byte test accepting 1 in 2048
//! two-byte heads (measured exhaustively over all 65 536, not sampled) and so is
//! not a claim to make about 8 GiB. See `slot_packer::carries_file_wide`. The
//! extension table covers the whole file at once and so has no such gap, which is
//! why `.pack` / `.idx` belong there too.
//!
//! ## The git case, precisely
//!
//! Two different things are called "a git object" and they need opposite
//! answers:
//!
//! * a **loose object file** in `.git/objects/ab/cdef…` is a zlib stream, has no
//!   extension at all, and must be stored raw. [`is_zlib_stream`] catches it.
//! * a `znippy-plugin-git` archive entry is the object's **canonical form**
//!   (`"<type> <size>\0<content>"`) — inflated bytes that compress perfectly
//!   well and *must* be compressed. It begins `commit `/`tree `/`blob `/`tag `,
//!   matches no probe there, and is correctly left alone.
//!
//! A rule of the shape "an oid-keyed entry is already compressed" would get the
//! second case backwards and silently store compressible data raw. Probing the
//! bytes at offset 0 gets both right. Both halves are asserted against a real
//! encoder in `ldeflate::precompressed`'s own tests.

use std::path::Path;

/// The probe itself, re-exported so `znippy_common::precompressed::*` keeps
/// naming exactly what it always named. These are `ldeflate`'s items, not copies
/// of them — breaking one there breaks every znippy caller, which is the point
/// of the move.
pub use ldeflate::precompressed::{ContentHint, SNIFF_PREFIX_LEN, is_zlib_stream, looks_compressed};

/// A batch-level compression policy: one [`ContentHint`] covering everything in
/// a run, plus the fallbacks for when the hint says nothing.
///
/// Batch-level because that is how the knowledge actually arrives — a caller
/// sealing a pack knows it for the whole pack, not entry by entry.
///
/// The two decision points are deliberately split, because the compress path has
/// two moments with different information available:
///
/// * [`skip_by_path`](Self::skip_by_path) runs during enumeration, before a
///   single byte is read;
/// * [`skip_by_bytes`](Self::skip_by_bytes) refines it inside the compressor,
///   once a chunk's real bytes are in hand, and is
///   `ldeflate::precompressed::SkipPolicy::skip` under a name that says which of
///   the two moments it belongs to.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct SkipPolicy(ldeflate::precompressed::SkipPolicy);

impl SkipPolicy {
    /// No claim: decide per entry from its name, then from its bytes.
    pub const fn resolve() -> Self {
        Self(ldeflate::precompressed::SkipPolicy::resolve())
    }

    /// Everything in this batch is already compressed — store it all raw.
    pub const fn already_compressed() -> Self {
        Self(ldeflate::precompressed::SkipPolicy::already_compressed())
    }

    /// Compress everything, whatever it looks like. The meaning of `--no-skip`.
    pub const fn compress_everything() -> Self {
        Self(ldeflate::precompressed::SkipPolicy::compress_everything())
    }

    /// Bridge from the historical `no_skip: bool`, which is precisely the
    /// two-valued subset of [`ContentHint`] that the CLI already exposed.
    pub const fn from_no_skip(no_skip: bool) -> Self {
        if no_skip { Self::compress_everything() } else { Self::resolve() }
    }

    pub const fn from_hint(hint: ContentHint) -> Self {
        Self(ldeflate::precompressed::SkipPolicy::from_hint(hint))
    }

    pub const fn hint(&self) -> ContentHint {
        self.0.hint()
    }

    /// The same policy as `ldeflate` sees it — for a caller handing bytes
    /// straight to `ldeflate::compress_objects` rather than to znippy's packer,
    /// so the two cannot disagree about what a hint means.
    pub const fn as_ldeflate(&self) -> ldeflate::precompressed::SkipPolicy {
        self.0
    }

    /// The decision available before any bytes are read: hint, else the
    /// extension table.
    ///
    /// This is the leg `ldeflate` cannot have — it is handed `&[u8]`, never a
    /// `Path`.
    ///
    /// `true` means "store raw". A `false` here is **not** final — the caller
    /// should still offer the bytes to [`skip_by_bytes`](Self::skip_by_bytes)
    /// once it has them.
    pub fn skip_by_path(&self, path: &Path) -> bool {
        match self.0.hint() {
            ContentHint::AlreadyCompressed => true,
            ContentHint::Compressible => false,
            ContentHint::Unknown => crate::index::is_probably_compressed(path),
        }
    }

    /// The refinement once real bytes are in hand: a magic-byte probe.
    ///
    /// Only meaningful when [`skip_by_path`](Self::skip_by_path) returned
    /// `false`; calling it otherwise is harmless but pointless. Honours the hint
    /// in both directions, so a `Compressible` batch is never vetoed by a probe.
    pub fn skip_by_bytes(&self, prefix: &[u8]) -> bool {
        self.0.skip(prefix)
    }
}

impl From<ldeflate::precompressed::SkipPolicy> for SkipPolicy {
    fn from(p: ldeflate::precompressed::SkipPolicy) -> Self {
        Self(p)
    }
}

impl From<SkipPolicy> for ldeflate::precompressed::SkipPolicy {
    fn from(p: SkipPolicy) -> Self {
        p.0
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    // The probe's own behaviour — the magic table, the bzip2/WebP special cases,
    // the zlib header against streams from a real encoder, canonical git object
    // bytes — is asserted in `ldeflate::precompressed`, which is where the code
    // now is. Repeating it here would be the twin this change removed.
    //
    // What is asserted here is what this crate adds: the path leg, the bridge
    // from `no_skip`, and — first, because everything else assumes it — that the
    // re-export really does reach `ldeflate` rather than a leftover copy.

    /// The dedup itself. If `znippy-common` ever grows a second magic table
    /// again, these calls stop agreeing.
    #[test]
    fn the_probe_is_ldeflates_and_not_a_second_copy() {
        for bytes in [
            b"PACK\x00\x00\x00\x02".as_slice(),
            b"\x1f\x8b\x08\x00".as_slice(),
            b"\x28\xb5\x2f\xfd\x00\x58".as_slice(),
            b"RIFF\x24\x00\x00\x00WEBPVP8 ".as_slice(),
            b"\x78\x9c".as_slice(),
            b"SELECT * FROM njord WHERE id = 42;".as_slice(),
            b"x = 1;\n".as_slice(),
            b"".as_slice(),
        ] {
            assert_eq!(
                looks_compressed(bytes),
                ldeflate::precompressed::looks_compressed(bytes),
                "the re-export must BE ldeflate's probe: {bytes:?}"
            );
            assert_eq!(
                is_zlib_stream(bytes),
                ldeflate::precompressed::is_zlib_stream(bytes),
                "the re-export must BE ldeflate's zlib probe: {bytes:?}"
            );
            // And the policy's byte leg must be the same decision, not a
            // parallel one.
            assert_eq!(
                SkipPolicy::resolve().skip_by_bytes(bytes),
                ldeflate::precompressed::SkipPolicy::resolve().skip(bytes),
                "skip_by_bytes must delegate: {bytes:?}"
            );
        }
        assert_eq!(SNIFF_PREFIX_LEN, ldeflate::precompressed::SNIFF_PREFIX_LEN);
    }

    /// The wrapper must not lose the hint on the way in or out — the failure
    /// that would make `--no-skip` silently stop meaning anything.
    #[test]
    fn the_hint_survives_the_wrapper_in_both_directions() {
        for (p, want) in [
            (SkipPolicy::resolve(), ContentHint::Unknown),
            (SkipPolicy::already_compressed(), ContentHint::AlreadyCompressed),
            (SkipPolicy::compress_everything(), ContentHint::Compressible),
        ] {
            assert_eq!(p.hint(), want);
            assert_eq!(p.as_ldeflate().hint(), want, "the ldeflate view must carry the same hint");
            assert_eq!(SkipPolicy::from(p.as_ldeflate()), p, "round-trip through ldeflate");
            assert_eq!(ldeflate::precompressed::SkipPolicy::from(p), p.as_ldeflate());
            assert_eq!(SkipPolicy::from_hint(want), p, "from_hint must agree with the constructor");
        }
        assert_eq!(SkipPolicy::default(), SkipPolicy::resolve(), "the probe is the default");
    }

    // ── the hint, both directions ────────────────────────────────────────────

    #[test]
    fn an_already_compressed_hint_is_honoured_without_inspection() {
        let p = SkipPolicy::already_compressed();
        // Plain compressible bytes under a plain name: only the hint can decide
        // this, and it must.
        assert!(p.skip_by_path(Path::new("objects/00ff")), "hint must win on the path");
        assert!(p.skip_by_bytes(b"SELECT * FROM njord;"), "hint must win on the bytes");
    }

    #[test]
    fn an_unhinted_batch_of_compressible_data_is_still_compressed() {
        // The other direction: a hint that swallows everything is as broken as
        // one that is ignored.
        let p = SkipPolicy::resolve();
        assert!(!p.skip_by_path(Path::new("dbdump/njord.dump")));
        assert!(!p.skip_by_bytes(b"SELECT * FROM njord WHERE id = 42;"));
    }

    #[test]
    fn a_compressible_hint_overrules_both_fallbacks() {
        let p = SkipPolicy::compress_everything();
        assert!(!p.skip_by_path(Path::new("photo.png")), "--no-skip must beat the extension table");
        assert!(!p.skip_by_bytes(b"\x89PNG\r\n\x1a\n"), "--no-skip must beat the probe");
    }

    #[test]
    fn no_skip_maps_onto_the_hint() {
        assert_eq!(SkipPolicy::from_no_skip(true), SkipPolicy::compress_everything());
        assert_eq!(SkipPolicy::from_no_skip(false), SkipPolicy::resolve());
    }

    /// The whole point of a sniffer: a name that lies.
    #[test]
    fn an_extension_that_lies_is_caught_by_the_bytes() {
        let p = SkipPolicy::resolve();
        let notes = Path::new("release-notes.txt");
        let zstd_bytes = b"\x28\xb5\x2f\xfd\x00\x58\x2d\x00\x00";

        assert!(!p.skip_by_path(notes), "the name says text, so the fast path lets it through");
        assert!(p.skip_by_bytes(zstd_bytes), "the bytes say zstd, and the bytes are the authority");
    }

    /// And the same trap in reverse: a `.pack` name over compressible bytes is
    /// still skipped by the fast path. Documented so the asymmetry is a decision
    /// rather than a surprise — the extension table is trusted, by design.
    #[test]
    fn the_extension_fast_path_covers_a_whole_file_not_just_its_head() {
        let p = SkipPolicy::resolve();
        assert!(p.skip_by_path(Path::new("objects/pack/pack-9f2c.pack")));
        assert!(p.skip_by_path(Path::new("objects/pack/pack-9f2c.idx")));
    }

    /// A git loose object is the case with **no extension at all**, so only the
    /// byte leg can reach it — and the path leg must say "not from the name",
    /// which is what makes the byte leg necessary rather than decorative.
    #[test]
    fn an_oid_named_loose_object_is_reached_only_by_the_bytes() {
        let p = SkipPolicy::resolve();
        let oid = Path::new(".git/objects/9f/2c1e0b7a4d3f8c6e5a2b9d0c7f4e1a8b3d6c9e2f");
        assert!(!p.skip_by_path(oid), "an oid has no extension for the table to match");
        assert!(p.skip_by_bytes(b"\x78\x01\x4b\xca\x49"), "so the zlib header must catch it");
    }
}