znippy-cli 0.2.4

CLI for Znippy, a parallel chunked compression system.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


![znippy](https://github.com/user-attachments/assets/7db1c1c1-d577-4f87-bfe1-11af6e8c58a0)

# Znippy
Znippy archive format based on Zstandard and Apache Arrow . Built for speed, streaming, and random access, it was originally created for Artisan—the airgap-driven artifact toolbox—and now stands as a fast, interchangeable backend.  

|                 | ⚡ Znippy                                                    | 🐢 tar + zstd                      |
| --------------- | ----------------------------------------------------------- | ---------------------------------- |
| ⚡ Compression   | 44.6 seconds<br>2767 MB → 911 MB<br>418 % compression ratio | 63.3 seconds<br>2767 MB → 722 MB   |
| ⚡ Decompression | 1.9 seconds<br>14979 chunks<br>fully indexed                | 3.9 seconds<br>streamed extraction |



# Snippy is dead — long live Znippy!  
The name Snippy was already in use in the genomics world, so we’re moving forward with Znippy — a name that proudly reflects its foundation in Zstandard (Zstd) and its focus on real-time, high-performance compression. 



## 🧩 Znippy Compression Pipeline – Visual Overview

```mermaid
flowchart LR
    %% Reader + ChunkRevolver
    Reader[Reader Thread] --> Revolver((ChunkRevolver))

    %% Fan-out
    Revolver --> C0[Compressor 0]
    Revolver --> C1[Compressor 1]
    Revolver --> C2[...]
    Revolver --> C31[Compressor 31]

    %% Fan-in
    C0 --> Writer[Writer Thread]
    C1 --> Writer
    C2 --> Writer
    C31 --> Writer

    %% Output and Index
    Writer --> Index[Arrow Index - znippy]
    Writer --> Zdata[zdata File]

    %% Microchunk entries in index
    Index --> M0[Microchunk 0]
    Index --> M1[Microchunk 1]
    Index --> M2[...]
    Index --> M63[Microchunk 63]

    M0 --> Zdata
    M1 --> Zdata
    M2 --> Zdata
    M63 --> Zdata

    %% Final archive
    Index --> Final[Znippy Archive Output]
    Zdata --> Final