znippy-cli 0.5.0

CLI for Znippy, a parallel chunked compression system.
docs.rs failed to build znippy-cli-0.5.0
Please check the build logs for more information.
See Builds for ideas on how to fix a failed build, or Metadata for how to configure docs.rs builds.
If you believe this is docs.rs' fault, open an issue.
Visit the last successful build: znippy-cli-0.2.5

znippy

Znippy

High-performance archive format with per-file compression, parallel processing, and random access. Built on Apache Arrow IPC + OpenZL (zstd+lz4 under the hood).

Benchmarks (32-core, OpenZL)

Test In Out Ratio Compress Decompress
text 500MB 500 MB 0.11 MB 4668x 1,493 MB/s 2,941 MB/s
single file 2GB 2,048 MB 0.40 MB 5095x 3,483 MB/s 3,127 MB/s
100k small files 977 MB 16.9 MB 57.8x 3,071 MB/s 769 MB/s
Rust deps (41k files) 988 MB 137 MB 7.2x 67.6 MB/s 1,337 MB/s
Java raw (191k files) 1,236 MB 444 MB 2.8x 83.0 MB/s 526 MB/s
rust crates (53k files) 1,298 MB 174 MB 7.5x 41 MB/s 1,417 MB/s

Architecture

flowchart LR
    subgraph Input
        Files[Input Files]
    end

    subgraph "Reader Thread"
        Reader[Reader] --> Split[Split into chunks<br/>10MB default]
        Split --> Revolver((ChunkRevolver))
    end

    subgraph "Compressor Threads (1..N cores)"
        Revolver --> C0[Core 0<br/>OpenZL + blake3]
        Revolver --> C1[Core 1<br/>OpenZL + blake3]
        Revolver --> C2[Core 2<br/>...]
        Revolver --> CN[Core N<br/>OpenZL + blake3]
    end

    subgraph "Writer Thread"
        C0 --> Writer[Writer]
        C1 --> Writer
        C2 --> Writer
        CN --> Writer
    end

    subgraph Output
        Writer --> Archive[".znippy<br/>(Arrow IPC index +<br/>checksums + config)"]
        Writer --> Zdata[".zdata<br/>(compressed chunks)"]
    end

    Files --> Reader

Decompression Pipeline

flowchart LR
    subgraph Input
        Index[".znippy index"] --> Reader
        Zdata[".zdata"] --> Reader
    end

    subgraph "Reader Thread"
        Reader[Reader<br/>seek + read chunks]
        Reader --> Revolver((ChunkRevolver))
    end

    subgraph "Decompressor Threads (1..N cores)"
        Revolver --> D0[Core 0<br/>zstd decompress]
        Revolver --> D1[Core 1<br/>zstd decompress]
        Revolver --> D2[Core 2<br/>...]
        Revolver --> DN[Core N<br/>zstd decompress]
    end

    subgraph "Writer Thread"
        D0 --> Writer[Writer<br/>+ blake3 verify]
        D1 --> Writer
        D2 --> Writer
        DN --> Writer
    end

    Writer --> Output[Restored Files]

Features

  • Parallel compression: fan-out to all cores via ChunkRevolver
  • Blake3 checksums: per-group integrity verification (machine-independent)
  • Arrow IPC index: queryable by DuckDB, Polars, DataFusion
  • Skip detection: already-compressed files (.zip, .gz, .png, etc.) stored as-is
  • Random access: seek directly to any file's chunks via index

Usage

# Compress a directory
znippy compress --input ./mydata --output archive.znippy

# Decompress
znippy decompress --input archive.znippy --output ./restored

# Verify integrity (no file writes)
znippy verify --input archive.znippy

# List contents
znippy list --input archive.znippy

Roadmap

  • v0.3.0 (current): OpenZL backend, plugin system (WASM + native), ZnippyArchive API
  • v0.4: Single-file format (Arrow IPC with inline binary column per chunk)
  • v0.5: Per-file-type specialized compression via plugins

Fan arts

znippy