zexc 1.1.2

zex native lossless compressor — CLI zexc, library zexc, files *.zex
Documentation
# ZEX Architecture

This document explains **how ZEX is built** so anyone can navigate the code.

**License:** project is **GPL-3.0-only** — see [`../LICENSE`](../LICENSE).

## Big picture

```
┌─────────────────────────────────────────────────────────────────┐
│                         User / CLI (`zexc`)                        │
│  compress | decompress | info   (+ optional password)            │
└────────────────────────────┬────────────────────────────────────┘
                             │
         ┌───────────────────┼───────────────────┐
         ▼                   ▼                   ▼
   One-shot API         Streaming API        Frame inspect
   compress()           StreamEncoder        FrameReader
   decompress()         StreamDecoder        ZexHeader
         │                   │
         └─────────┬─────────┘
                   ▼
         ┌─────────────────────┐
         │   Frame container   │  magic ZEX1, header, blocks, CRC
         │   format/*          │  optional AEAD shell
         └──────────┬──────────┘
                    ▼
         ┌─────────────────────┐
         │  Native codec       │  algorithms/zex_native.rs
         │  (per block)        │  hash chain · lazy · Huffman
         └──────────┬──────────┘
                    ▼
         ┌─────────────────────┐
         │  utils/             │  simd · entropy · crc · security
         └─────────────────────┘
```

ZEX is **one original algorithm**, not a wrapper that calls LZ4 or Zstd libraries.

## Crate layout

| Path | Role |
|------|------|
| `src/lib.rs` | Public API, errors, re-exports |
| `src/main.rs` | CLI binary `zexc` |
| `src/cli/` | Clap argument definitions |
| `src/core/encoder.rs` | One-shot frame encode (+ AEAD) |
| `src/core/decoder.rs` | One-shot frame decode (+ AEAD) |
| `src/core/stream.rs` | Incremental `Write` / `Read` API |
| `src/core/frame.rs` | `zex info` scanning |
| `src/core/dictionary.rs` | Dictionary train stub + id |
| `src/algorithms/zex_native.rs` | **The compressor/decompressor** |
| `src/format/header.rs` | Frame header wire format |
| `src/format/block.rs` | Block header wire format |
| `src/utils/simd.rs` | Wide `match_len`, RLE copy |
| `src/utils/entropy.rs` | Canonical Huffman |
| `src/utils/bitstream.rs` | Bit packer for Huffman |
| `src/utils/crc.rs` | CRC32 (content integrity) |
| `src/utils/security.rs` | Size caps / match bounds |
| `src/crypto/` | ChaCha20-Poly1305 + BLAKE3 KDF |
| `docs/` | Human documentation (this tree) |
| `benches/` | Criterion microbenchmarks |
| `scripts/` | Compare vs system `zstd` / `lz4` |

## Data path (compress)

```
plaintext bytes
    │
    ├─► split into independent blocks (default 256 KiB)
    │
    ▼  each block (zex_native)
 match finder (hash table + optional chain / dual hash / lazy)
    │
    ▼
 sequences: (literal_run, match_offset, match_len)
    │
    ▼
 literals: raw  OR  Huffman if smaller
    │
    ▼
 block payload bytes
    │
    ▼
 BlockHeader { type, unc_size, comp_size } + payload
    │
    ▼
 [optional: collect all blocks → AEAD encrypt]
    │
    ▼
 ZexHeader + body + CRC32(plaintext)
```

## Data path (decompress)

```
.zex file
    │
    ▼
 parse ZexHeader (magic, flags, sizes)
    │
    ├─ if ENCRYPTED → salt/nonce + open ChaCha20-Poly1305
    │
    ▼
 for each block:
    Raw  → copy
    Zex  → expand sequences (literals + match copies)
    End  → stop
    │
    ▼
 verify CRC32 if flag set
```

## Profiles vs levels

| Profile | Meaning |
|---------|---------|
| **Turbo** | Cap effective level low → shallow search, high skip |
| **Balanced** | Use user level as-is |
| **Max** | Floor effective level higher → more search + entropy |

Levels **1–22** inside `params_for_level()` control:

- hash table size  
- chain depth / probes  
- lazy matching  
- dual 4/5-byte hash  
- Huffman on literals  
- window size  

## Security layers

1. **Safe Rust** codec (no UB in match copy path)  
2. **SecurityPolicy** hard caps (block size, total output)  
3. **CRC32** default integrity of plaintext  
4. **Optional AEAD** (confidentiality + auth of compressed body)  
5. Decoder rejects bad offsets, absurd sequence counts  

## Streaming design choices

- Blocks are **independent** → encoder can emit a block as soon as `block_size` is full.  
- Header stores `content_size = unknown`.  
- AEAD of the whole body needs all ciphertext up front → **not** on stream API.  
- See [STREAMING.md]STREAMING.md.

## What we learned from LZ4 / Zstd (ideas only)

| Idea | Source family | ZEX use |
|------|---------------|---------|
| Hash table + acceleration skip | LZ4 | Turbo levels |
| Hash chains / lazy | Zstd | Mid–high levels |
| Literals + sequences model | Zstd | Native sequences |
| Levels 1–22 scale | Zstd UX | Profile + level |

Wire formats are **not** compatible with `.lz4` or `.zst`.

## Feature flags

| Feature | Default | Effect |
|---------|---------|--------|
| `std` | yes | Hosted std environment |
| `simd` | yes | Wide match_len helpers |
| `encryption` | yes | AEAD + password KDF |


## Dictionary history (match finder)

Optional dictionary is a **virtual prefix** before each independent block:

```
[ dictionary bytes | block content … ]
         ↑                    ↑
   seeded into hash       compressed
   tables only            & stored
```

- **Encode:** match finder can copy from dict; offsets may point “before” block start  
- **Decode:** `copy_match_with_history` reads dict when `offset > out_len`  
- **Frame:** `dictionary_id` + `DICTIONARY` flag; decoder must supply same dict  
- Not sent inside the payload (both peers share dict out-of-band)

API: `CompressOptions::dictionary`, `DecompressOptions::dictionary`,
`Dictionary::train_simple` / `from_bytes`.

## Further reading

- [FORMAT.md]FORMAT.md — on-wire bytes  
- [STREAMING.md]STREAMING.md — stream API  
- [API.md]API.md — library usage  
- [PACKAGING.md]PACKAGING.md — install / release  
- [BENCHMARKS.md]BENCHMARKS.md — how we measure vs zstd/lz4