ldeflate 0.1.3

Pure Rust parallel DEFLATE/zlib compressor — the write side of linflate. Two-queue gatling scheduling: coalesce small objects, split large streams at full-flush boundaries.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102

//! `ldeflate` — pure-Rust **parallel DEFLATE/zlib compressor**. The write side
//! of [`linflate`].
//!
//! `linflate` reads DEFLATE fast; this crate writes it fast. The gap it closes
//! is concrete: serving a git packfile means DEFLATE-compressing thousands of
//! objects, and before this crate there was **no compression code anywhere in
//! the znippy-zoomies workspace** except the vendored C zstd in `zstd-sys-rs`.
//!
//! # What is actually parallel here
//!
//! Not the entropy coder — that is `zlib-rs`, reused unchanged (LAW 5). What is
//! new is the **scheduling**, and it has two axes chosen by size:
//!
//! - **Across objects.** A pack is thousands of independent streams. Small ones
//!   are *coalesced* into batches so one task drains many, because dispatching a
//!   task to deflate 200 bytes costs more than the deflate. This is the axis
//!   that matters for git, whose objects are overwhelmingly tiny.
//! - **Within one stream.** A large object is cut at `Z_FULL_FLUSH` boundaries —
//!   the same seam `linflate::deflate_scan::find_all_flushes` looks for on the
//!   read side — so one 400 MB blob cannot serialise a worker. This is the axis
//!   that matters for LFS-shaped content.
//!
//! Both run on one no-barrier `gatling` pass. No rayon (LAW 3); no private
//! thread pool; no barrier inside the stream.
//!
//! # Guarantees
//!
//! - **Stock-`git`-readable.** Output is ordinary RFC-1950 zlib. The acceptance
//!   test in `tests/stock_git.rs` builds a real repository whose entire object
//!   store this crate wrote, then runs `git fsck --strict` and `git index-pack
//!   --strict` over a real packfile and byte-compares every object back.
//! - **Pure Rust.** `flate2` is pinned to `default-features = false,
//!   features = ["zlib-rs"]`. No C, no cmake, no bindgen, no `build.rs`, no
//!   build-time network. `tests/no_c_toolchain.rs` asserts it mechanically.
//! - **Deterministic.** Same input + same [`Config`] ⇒ byte-identical output at
//!   any worker count, because the work plan is a pure function of the object
//!   sizes and the config and never reads the core count.
//!
//! # Example
//!
//! ```
//! use ldeflate::{compress_objects, Config};
//!
//! let a = b"tree 42\0...".to_vec();
//! let b = vec![b'x'; 4_000_000];
//! let objects: Vec<&[u8]> = vec![&a, &b];
//!
//! let streams = compress_objects(&objects, &Config::default()).unwrap();
//! assert_eq!(streams.len(), 2);
//! // Each element is a complete zlib stream; `git` reads them verbatim.
//! ```
//!
//! [`linflate`]: https://crates.io/crates/linflate

pub mod container;
pub mod crc32;
pub mod gzip;
pub mod limits;
pub mod plan;
pub mod precompressed;
pub mod split;
pub mod stream;
pub mod unit;
pub mod zip;

pub use container::Container;
pub use crc32::{crc32, crc32_combine};
pub use gzip::{gzip_compress, gzip_compress_members, gzip_members_of};
pub use limits::{MAX_PIECES, check_pieces};
// The whole probe surface, not two thirds of it: `is_zlib_stream` decides the
// git loose-object case on its own and `SNIFF_PREFIX_LEN` is how a caller sizes
// the head it hands in, so both are part of the contract downstream reaches for.
pub use plan::{
    Config, Unit, compress_objects, compress_objects_checksummed, compress_objects_serial, plan,
};
pub use precompressed::{
    ContentHint, SNIFF_PREFIX_LEN, SkipPolicy, is_zlib_stream, looks_compressed,
};
pub use split::{plan_segments, split_compress_serial};
pub use stream::{Placed, compress_objects_to_writer, compute_workers};
pub use unit::{
    DeflateError, Encoder, WINDOW_BITS_DEFAULT, WINDOW_BITS_MIN, adler32, adler32_combine,
    deflate_compress, zlib_compress,
};
pub use zip::{Entry as ZipEntry, jar_write, zip_write, zip_write_placed};

/// Compress one buffer, splitting it across workers at full-flush boundaries.
///
/// The single-object form of the within-stream axis: use it for one large blob
/// (an LFS object, a big packed tree) when there is no batch of siblings to fan
/// out over instead.
pub fn compress_split(src: &[u8], cfg: &Config) -> Result<Vec<u8>, DeflateError> {
    let one: [&[u8]; 1] = [src];
    let cfg = Config {
        split_above: 1,
        coalesce_below: 0,
        ..*cfg
    };
    Ok(compress_objects(&one, &cfg)?
        .pop()
        .expect("one object in, one out"))
}