ldeflate

Pure-Rust parallel DEFLATE/zlib compressor — the write side of linflate.

linflate reads DEFLATE fast. This writes it fast. It exists because serving or repacking a git packfile means DEFLATE-compressing thousands of objects, and before this crate there was no compression code anywhere in the znippy-zoomies workspace.

use ldeflate::{compress_objects, Config};

let objects: Vec<&[u8]> = vec![b"tree 42\0...", b"blob..."];
let streams = compress_objects(&objects, &Config::default())?;
// Each element is a complete RFC-1950 zlib stream. `git` reads them verbatim.
# Ok::<(), ldeflate::DeflateError>(())

What is parallel

Not the entropy coder — that is zlib-rs, reused unchanged. The scheduling is new, and it has two axes chosen per object by size, on one no-barrier gatling pass:

small objects are coalesced into batches, so one task drains many and the expensive per-stream compressor setup is paid once per batch rather than once per object;
large objects are split at Z_FULL_FLUSH boundaries, so a single big blob cannot serialise a worker.

Git objects are 93.8% under 64 KiB (measured, 48 917 real objects), so they take the first path. LFS objects are megabytes, so they take the second. The same Config serves both.

Guarantees

Stock git accepts the output. tests/stock_git.rs builds a repository whose entire object store this crate wrote and runs git fsck --full --strict over it, then assembles a real pack-v2 and runs git index-pack --strict. Both have negative controls proving the checks can fail.
Pure Rust. flate2 is pinned to default-features = false, features = ["zlib-rs"]. No C, no cmake, no bindgen, no build.rs, no build-time network — asserted mechanically by tests/no_c_toolchain.rs against the resolved dependency graph.
Deterministic. Byte-identical output at any worker count: the work plan is a pure function of the object sizes and the Config, and never reads the core count.
rayon-free (constellation LAW 3). All threading routes through gatling.

Measured, oden (Threadripper PRO 3975WX, 32 cores), 2026-08-03


parallel cost, ~49 k real git objects, ~1.68 GiB, level 6	1.025×–1.054× the CPU of serial at 32 threads — n=7 runs, reproducible to 2.8%
efficiency vs cores actually held	94.9%–97.6% at 32 threads, over occupancies from 10.9 to 24.3 of 32 cores held
sustained	2136 MB/s across 39 consecutive passes, median 100 ms window fully saturated
ratio cost of coalescing	zero — not one output byte changes
ratio cost of splitting	2.2% at the default 1 MiB segment, only on objects ≥ 4 MiB
skipping already-compressed input	≈20× less CPU (94.8%–95.6% saved) for +0.41% size, on 24 real packfiles

Why no headline “×N faster” figure. There was one — 28.30× at 32 threads — and a 7-run re-measurement on 2026-08-03 retired it as a crate claim: over those runs the 32-thread speedup ranged 10.38× to 23.20× purely with how many cores the box would give us, while the CPU cost of the schedule held to 1.025–1.054×. 28.30× was never contradicted (it is consistent with, and slightly conservative against, the re-runs), but it is unreproducible on a shared machine, so it is not quoted here as a property of the code. On a genuinely idle 32-core box the re-measured overhead predicts 30.4×–31.2×; that prediction is untested — oden has not been idle. See Finding 4b in the design doc.

Full tables, the two hypotheses these sweeps falsified, and the honest caveats: ../.nornir/ldeflate-design.md. The engine patterns it builds on, written down independently: ../.nornir/gatling-tricks.md.

When NOT to use it

Serving a clone. The bytes are already deflated in the pack the client pushed; copying them is a memcpy and no compressor beats that.
Large already-compressed content (most LFS objects: video, images, archives). Level-6 DEFLATE measured a compression ratio of 1.000 on high-entropy data — literally nothing — while running 42× slower than a SHA-1 pass over the same bytes. Route that content to store-raw before it reaches a compressor.

License