# ZEX Architecture
This document explains **how ZEX is built** so anyone can navigate the code.
**License:** project is **GPL-3.0-only** — see [`../LICENSE`](../LICENSE).
## Big picture
```
┌─────────────────────────────────────────────────────────────────┐
│ User / CLI (`zexc`) │
│
┌───────────────────┼───────────────────┐
▼ ▼ ▼
One-shot API Streaming API Frame inspect
compress() StreamEncoder FrameReader
decompress() StreamDecoder ZexHeader
│ │
└─────────┬─────────┘
▼
┌─────────────────────┐
│ Frame container │ magic ZEX1, header, blocks, CRC
│ format/* │ optional AEAD shell
└──────────┬──────────┘
▼
┌─────────────────────┐
│ Native codec │ algorithms/zex_native.rs
│ (per block) │ hash chain · lazy · Huffman
└──────────┬──────────┘
▼
┌─────────────────────┐
│ utils/ │ simd · entropy · crc · security
└─────────────────────┘
```
ZEX is **one original algorithm**, not a wrapper that calls LZ4 or Zstd libraries.
## Crate layout
| `src/lib.rs` | Public API, errors, re-exports |
| `src/main.rs` | CLI binary `zexc` |
| `src/cli/` | Clap argument definitions |
| `src/core/encoder.rs` | One-shot frame encode (+ AEAD) |
| `src/core/decoder.rs` | One-shot frame decode (+ AEAD) |
| `src/core/stream.rs` | Incremental `Write` / `Read` API |
| `src/core/frame.rs` | `zex info` scanning |
| `src/core/dictionary.rs` | Dictionary train stub + id |
| `src/algorithms/zex_native.rs` | **The compressor/decompressor** |
| `src/format/header.rs` | Frame header wire format |
| `src/format/block.rs` | Block header wire format |
| `src/utils/simd.rs` | Wide `match_len`, RLE copy |
| `src/utils/entropy.rs` | Canonical Huffman |
| `src/utils/bitstream.rs` | Bit packer for Huffman |
| `src/utils/crc.rs` | CRC32 (content integrity) |
| `src/utils/security.rs` | Size caps / match bounds |
| `src/crypto/` | ChaCha20-Poly1305 + BLAKE3 KDF |
| `docs/` | Human documentation (this tree) |
| `benches/` | Criterion microbenchmarks |
| `scripts/` | Compare vs system `zstd` / `lz4` |
## Data path (compress)
```
plaintext bytes
│
├─► split into independent blocks (default 256 KiB)
│
▼ each block (zex_native)
match finder (hash table + optional chain / dual hash / lazy)
│
▼
sequences: (literal_run, match_offset, match_len)
│
▼
literals: raw OR Huffman if smaller
│
▼
block payload bytes
│
▼
BlockHeader { type, unc_size, comp_size } + payload
│
▼
[optional: collect all blocks → AEAD encrypt]
│
▼
ZexHeader + body + CRC32(plaintext)
```
## Data path (decompress)
```
.zex file
│
▼
parse ZexHeader (magic, flags, sizes)
│
├─ if ENCRYPTED → salt/nonce + open ChaCha20-Poly1305
│
▼
for each block:
Raw → copy
Zex → expand sequences (literals + match copies)
End → stop
│
▼
verify CRC32 if flag set
```
## Profiles vs levels
| **Turbo** | Cap effective level low → shallow search, high skip |
| **Balanced** | Use user level as-is |
| **Max** | Floor effective level higher → more search + entropy |
Levels **1–22** inside `params_for_level()` control:
- hash table size
- chain depth / probes
- lazy matching
- dual 4/5-byte hash
- Huffman on literals
- window size
## Security layers
1. **Safe Rust** codec (no UB in match copy path)
2. **SecurityPolicy** hard caps (block size, total output)
3. **CRC32** default integrity of plaintext
4. **Optional AEAD** (confidentiality + auth of compressed body)
5. Decoder rejects bad offsets, absurd sequence counts
## Streaming design choices
- Blocks are **independent** → encoder can emit a block as soon as `block_size` is full.
- Header stores `content_size = unknown`.
- AEAD of the whole body needs all ciphertext up front → **not** on stream API.
- See [STREAMING.md](STREAMING.md).
## What we learned from LZ4 / Zstd (ideas only)
| Hash table + acceleration skip | LZ4 | Turbo levels |
| Hash chains / lazy | Zstd | Mid–high levels |
| Literals + sequences model | Zstd | Native sequences |
| Levels 1–22 scale | Zstd UX | Profile + level |
Wire formats are **not** compatible with `.lz4` or `.zst`.
## Feature flags
| `std` | yes | Hosted std environment |
| `simd` | yes | Wide match_len helpers |
| `encryption` | yes | AEAD + password KDF |
| `legacy-backends` | no | Old experimental modules |
## Dictionary history (match finder)
Optional dictionary is a **virtual prefix** before each independent block:
```
[ dictionary bytes | block content … ]
↑ ↑
seeded into hash compressed
tables only & stored
```
- **Encode:** match finder can copy from dict; offsets may point “before” block start
- **Decode:** `copy_match_with_history` reads dict when `offset > out_len`
- **Frame:** `dictionary_id` + `DICTIONARY` flag; decoder must supply same dict
- Not sent inside the payload (both peers share dict out-of-band)
API: `CompressOptions::dictionary`, `DecompressOptions::dictionary`,
`Dictionary::train_simple` / `from_bytes`.
## Further reading
- [FORMAT.md](FORMAT.md) — on-wire bytes
- [STREAMING.md](STREAMING.md) — stream API
- [API.md](API.md) — library usage
- [PACKAGING.md](PACKAGING.md) — install / release
- [BENCHMARKS.md](BENCHMARKS.md) — how we measure vs zstd/lz4