znippy-common 0.9.13

Core logic and data structures for Znippy, a parallel chunked compression system.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

<!-- ⚠ GENERATED by nornir from .nornir/README.md — DO NOT EDIT this file -->

![znippy](https://github.com/user-attachments/assets/7db1c1c1-d577-4f87-bfe1-11af6e8c58a0)

# znippy

> A parallel, random-access archive: pack a directory at all-core speed, pull any
> single file back without unpacking the rest — and query the index in
> **DuckDB / Polars / DataFusion**, because it's just Arrow IPC.

⚡ **GATLING** — one all-core engine · zero rayon · zero-copy · ~11/12 cores

## Mashup — one archive vs tar/zip/7z/parquet

znippy is the only archive that's all-core on *both* ends, random-access per file,
**and** directly queryable — without pretending to beat zstd/7z on raw ratio:

| capability | **znippy** | tar + zstd | zip | 7z | parquet |
|---|:--:|:--:|:--:|:--:|:--:|
| All-core **compress** | **✅** | ~ (`-T0`) | ◐ | ✗ | ✅ |
| All-core **decompress** | **✅** | ✗ serial | ✗ | ✗ | ✅ |
| Random single-file read | **✅** | ✗ | ✅ | ✗ (solid) | — (columnar) |
| **Query without the tool** (DuckDB / Polars) | **✅** | ✗ | ✗ | ✗ | ✅ |
| Best pure **ratio** | ◐ | ✅ (`-19`) | ✗ | ✅ | — |

*The full capability matrix and every measured per-backend throughput table (JAR /
ZIP / gzip / bzip2 decode vs the legacy C tools) are in the full overview.*

### Measured — znippy vs Arrow-IPC / zstd

One byte-identical mixed corpus (compressible text + incompressible blobs) packed
into a single compressed, queryable Arrow-IPC container two ways — znippy's
all-core pipeline vs a stock arrow-rs `FileWriter` with `CompressionType::ZSTD` —
then fully read back. Same box, same run, warmup + iterations:

<!--
  `benches name=mashup winner=row`, NOT a bare `mashup`. The prose above promises ONE
  head-to-head: `znippy_mashup` vs `arrowipc_mashup`, the two arms that pack the same
  corpus. A bare `mashup` auto-discovers "systems" from the leading `_`-delimited
  segment of every result name in the run, which here means it treats each of znippy's
  own ~20 arms (archive_…, footer_…, stree_…, subindex_…) as a separate rival system —
  it rendered a 67-column table with two populated cells per row. Filtering by name
  keeps this block the comparison the paragraph describes.
  See workspace_nordisk/nornir_agents_problems.md N2.
-->
<!-- nornir:gen:start:benches name=mashup winner=row -->
**v0.9.6 · oden · 32 cores · 2026-07-22**

| workload | compress_mbs | cores | decompress_mbs | files | in_mb | peak_rss_mb | ratio_x |
|---|---|---|---|---|---|---|---|
| arrowipc_mashup | **416.22** | 32 | 908.10 | 8000 | 1000 | 5208 | 1.44 |
| znippy_mashup | 111.89 | 32 | **8466** | 8000 | 1000 | 5208 | 1.47 |
<!-- nornir:gen:end:benches -->

*Not the same job: znippy stores every file as its own random-access, blake3-verified
chunk behind a queryable FST index and an immutable seal; the Arrow-IPC/zstd rival
concatenates ~1000 files per record-batch into one zstd stream — smaller granularity
traded away, no per-file addressing. So arrow's simpler batching wins raw
compress throughput while znippy wins decompress and keeps capabilities arrow can't
offer; both hit the same ratio on identical distinct data. Each side uses its
library's default codec settings.*

## Read the real docs

- 📖 **[The complete manual — `docs/manual.pdf`](docs/manual.pdf)** — every subsystem chapter (format, Gatling pipeline, random-access, Iceberg/seal, decompressors).
- 📄 **[Full overview — `.nornir/README-full.md`](.nornir/README-full.md)** — the detailed pitch, quickstart, what's-new, the full competitive matrix, and all measured benchmarks.
- 📝 [`docs/book.md`](docs/book.md) — the manual, as Markdown.