znippy-compress 0.9.5

Compression logic for Znippy, a parallel chunked compression system.
docs.rs failed to build znippy-compress-0.9.5
Please check the build logs for more information.
See Builds for ideas on how to fix a failed build, or Metadata for how to configure docs.rs builds.
If you believe this is docs.rs' fault, open an issue.
Visit the last successful build: znippy-compress-0.2.5

znippy

znippy

A parallel, random-access archive: pack a directory at all-core speed, pull any single file back without unpacking the rest — and query the index in DuckDB / Polars / DataFusion, because it's just Arrow IPC.

⚡ GATLING — one all-core engine · zero rayon · zero-copy · ~11/12 cores

Mashup — one archive vs tar/zip/7z/parquet

znippy is the only archive that's all-core on both ends, random-access per file, and directly queryable — without pretending to beat zstd/7z on raw ratio:

capability znippy tar + zstd zip 7z parquet
All-core compress ✅ ~ (-T0) ◐ ✗ ✅
All-core decompress ✅ ✗ serial ✗ ✗ ✅
Random single-file read ✅ ✗ ✅ ✗ (solid) — (columnar)
Query without the tool (DuckDB / Polars) ✅ ✗ ✗ ✗ ✅
Best pure ratio ◐ ✅ (-19) ✗ ✅ —

The full capability matrix and every measured per-backend throughput table (JAR / ZIP / gzip / bzip2 decode vs the legacy C tools) are in the full overview.

Measured — znippy vs Arrow-IPC / zstd

One byte-identical mixed corpus (compressible text + incompressible blobs) packed into a single compressed, queryable Arrow-IPC container two ways — znippy's all-core pipeline vs a stock arrow-rs FileWriter with CompressionType::ZSTD — then fully read back. Same box, same run, warmup + iterations:

v0.9.0 · threadripper-3975wx · 32 cores · 2026-06-18

footer_seal_l — 1.30 delta_pct · 1M rows · 0.58 seal_mrows_s (single-system bench; competitive matrix in README-full)

Not the same job: znippy stores every file as its own random-access, blake3-verified chunk behind a queryable FST index and an immutable seal; the Arrow-IPC/zstd rival concatenates ~1000 files per record-batch into one zstd stream — smaller granularity traded away, no per-file addressing. So arrow's simpler batching wins raw compress throughput while znippy wins decompress and keeps capabilities arrow can't offer; both hit the same ratio on identical distinct data. Each side uses its library's default codec settings.

Read the real docs