# ZEXC
**zexc** is a modern, pure-Rust lossless compression library and command-line tool. It implements an **original** codec and container format (`.zex`, magic `ZEX1`)—not a wrapper around LZ4 or Zstandard.
| **Author** | **Ali Zain** \<[alizain@zainiumdynamics.tech](mailto:alizain@zainiumdynamics.tech)\> |
| **Website** | [zainiumdynamics.tech](https://zainiumdynamics.tech) |
| **Language** | Rust 2024 (MSRV 1.85) |
| **License** | **[GPL-3.0-only](LICENSE)** — GNU GPLv3 **strict** (no “or later”) |
| **crates.io / crate** | **`zexc`** (`use zexc::…`) |
| **CLI binary** | **`zexc`** |
| **C library** | `libzexc` / `zexc.h`|
| **File extension** | **`.zex`** |
| **Version** | 1.2.0 (see [crates.io/crates/zexc](https://crates.io/crates/zexc)) |
---
## Highlights
- **Native algorithm** — hash-chain match finding, lazy matching, dual hashing, optional Huffman literals, and rANS sequence coding with repeat offsets
- **Performance profiles** — `turbo` (speed), `balanced` (default), `max` (ratio), levels **1–22**
- **SIMD-oriented hot paths** — wide `match_len` and fast RLE-aware copies
- **Streaming I/O** — `StreamEncoder` / `StreamDecoder` implementing `std::io::Write` / `Read`
- **Dictionary history** — shared dictionary as virtual prefix for small-message workloads
- **Optional AEAD** — ChaCha20-Poly1305 with BLAKE3 password-based key derivation
- **Safe defaults** — CRC32 content checksum, independent blocks, hard size caps
- **Production CLI** — compress, decompress, info, train-dict; stdin/stdout pipes via `-`
ZEX is designed for systems that need a **self-contained** compressor: auditable Rust, no `libzstd` / `liblz4` dependency, and a clear path from library API to CLI and packaging.
---
## Table of contents
1. [Installation](#installation)
2. [Command-line usage](#command-line-usage)
3. [Library usage](#library-usage)
4. [Language bindings](#language-bindings)
5. [Compression profiles](#compression-profiles)
6. [Security model](#security-model)
7. [Project layout](#project-layout)
8. [Documentation](#documentation)
9. [Building, testing, and benchmarks](#building-testing-and-benchmarks)
10. [Performance expectations](#performance-expectations)
11. [Roadmap](#roadmap)
12. [License](#license)
---
## Build configuration (`zexc.toml`)
Shared vs static vs CLI is **not** hard-coded in the Makefile. Edit **`zexc.toml`**:
```toml
[library]
shared = true # libzexc.so
static = true # libzexc.a
rlib = true
[binary]
standalone = true # CLI: zexc
[features]
simd = true
encryption = true
c-api = true
```
```bash
make configure # apply zexc.toml
make release # build
make show-config
```
See [docs/CONFIG.md](docs/CONFIG.md).
## Installation
### From source
```bash
git clone https://gitlab.com/alizain.arch/zexc.git
cd zexc
cargo build --release --features cli
# binary: target/release/zexc
```
### Install the CLI
From crates.io (after publish):
```bash
cargo install zexc
```
From source:
```bash
cargo install --path . --locked
# or
make install
```
### Feature flags
| `std` | yes | Standard library support |
| `simd` | yes | Wide-load match helpers |
| `encryption` | yes | ChaCha20-Poly1305 + password KDF |
| `cli` | yes | Build the `zexc` binary |
| `c-api` | yes | C ABI (`libzexc` / `zexc.h`) |
```bash
# Minimal build without encryption
cargo build --release --no-default-features --features std,simd
```
---
## Command-line usage
Primary form (positional input and output):
```bash
zexc compress input.txt output.zex
zexc decompress output.zex recovered.txt
zexc info file.zex
zexc test file.zex
```
```text
zexc <COMMAND> ...
Commands:
compress input output.zex
decompress input.zex output
info file.zex
test file.zex
train-dict out.dict samples...
bench file
```
### Options (common)
| `-l N` | Level 1–22 |
| `-m turbo\|balanced\|max` | Profile |
| `-e PASS` | Encrypt / decrypt password |
| `-D dict` | Dictionary file |
| `-T N` | Threads |
| `-q` | Quiet |
| `--rm` | Delete input after success |
### Pipes
```bash
```
### Dictionary
```bash
zexc train-dict http.dict samples/*.log
zexc compress msg.bin msg.zex -D http.dict
zexc decompress msg.zex msg.out -D http.dict
```
### Archive
```bash
zexc archive create bundle.zexa a.txt b.bin
zexc archive list bundle.zexa
zexc archive extract bundle.zexa ./out
```
## Library usage
Add the package to your `Cargo.toml`. crates.io name and import are both **`zexc`**.
```toml
[dependencies]
zexc = "1.1"
```
### One-shot API
```rust
use zexc::{compress, decompress, CompressOptions, CompressionMode};
fn main() -> zexc::Result<()> {
let opts = CompressOptions::default()
.mode(CompressionMode::Balanced)
.level(3)
.threads(true);
let compressed = compress(b"example payload", &opts)?;
let plain = decompress(&compressed)?;
assert_eq!(plain, b"example payload");
Ok(())
}
```
### Streaming API
```rust
use std::io::{Read, Write};
use zexc::{CompressOptions, StreamDecoder, StreamEncoder};
fn compress_stream(input: &[u8]) -> zexc::Result<Vec<u8>> {
let mut out = Vec::new();
let mut enc = StreamEncoder::new(&mut out, CompressOptions::default());
enc.write_all(input)?;
enc.finish()?;
Ok(out)
}
fn decompress_stream(zex_bytes: &[u8]) -> std::io::Result<Vec<u8>> {
let mut dec = StreamDecoder::new(std::io::Cursor::new(zex_bytes));
let mut plain = Vec::new();
dec.read_to_end(&mut plain)?;
Ok(plain)
}
```
See [docs/STREAMING.md](docs/STREAMING.md) for lifecycle rules (always call `finish()` on encode). Streaming does **not** support whole-frame AEAD; use the one-shot API for encrypted frames.
### Dictionary and encryption (library)
```rust
use zexc::{
compress_with_dict, decompress_with_options,
CompressOptions, DecompressOptions, Dictionary,
};
#[cfg(feature = "encryption")]
use zexc::EncryptionKey;
let dict = Dictionary::train_simple(&samples, 64 * 1024)?;
let opts = CompressOptions::default().level(6).dictionary(dict.clone());
#[cfg(feature = "encryption")]
let opts = opts.encrypt(EncryptionKey::Password("secret".into()));
let frame = compress_with_dict(message, &opts, dict.clone())?;
let plain = decompress_with_options(
&frame,
&DecompressOptions {
dictionary: Some(dict),
#[cfg(feature = "encryption")]
encryption: Some(EncryptionKey::Password("secret".into())),
..Default::default()
},
)?;
```
More examples: `examples/simple_compress.rs`, `examples/stream_compress.rs`, `examples/dictionary_train.rs`.
---
## Language bindings
| Python (PyO3 / maturin) | [`bindings/python`](bindings/python) | `pip install maturin && cd bindings/python && maturin develop --release` |
| Go (cgo, over the C ABI) | [`bindings/go`](bindings/go) | `cargo build --release --features c-api && cd bindings/go && go test ./...` |
Both wrap the same native codec — no libzstd/liblz4 anywhere in the call chain.
See each directory's README for the full API and licensing notes (GPLv3 applies
to distributed binaries/wheels linking `libzexc`, same as the C ABI).
---
## Compression profiles
| **Turbo** | `-m turbo` / `CompressionMode::Turbo` | Latency-sensitive paths; shallow search |
| **Balanced** | `-m balanced` (default) | General-purpose default |
| **Max** | `-m max` | Prefer ratio; deeper search and entropy |
Levels **1–22** scale hash table size, chain depth, lazy matching, dual-hash, literal Huffman, and sequence rANS. Exact parameters are documented in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
---
## Security model
| **Memory safety** | Codec implemented in safe Rust (no unsafe in the match/copy hot path) |
| **Integrity** | Optional but default **CRC32** over plaintext |
| **Confidentiality** | Optional **ChaCha20-Poly1305** over the compressed body |
| **DoS bounds** | `SecurityPolicy` caps on block size and total output |
| **Match validation** | Offsets and lengths checked against history (including dictionary) |
Encryption authenticates the compressed payload; a wrong password fails closed (authentication error). Dictionary id mismatches also fail closed.
Use `.secure()` / `--secure` for stricter absolute size limits.
---
## Project layout
```text
zexc/ (this tree may still be named zex-compressor on disk)
├── src/
│ ├── algorithms/zex_native.rs # core compressor / decompressor
│ ├── core/ # frame encode, decode, stream, dictionary
│ ├── format/ # header and block wire format
│ ├── utils/ # SIMD, Huffman, rANS/FSE, CRC, security
│ ├── crypto/ # AEAD + KDF (feature-gated)
│ ├── cli/ # clap definitions
│ ├── lib.rs
│ └── main.rs # zexc binary
├── include/zexc.h # C API
├── bindings/
│ ├── python/ # PyO3 / maturin
│ └── go/ # cgo, over the C API
├── docs/ # architecture, format, API, packaging
├── examples/
├── benches/
├── scripts/
├── fuzz/ # cargo-fuzz targets + seed corpus
├── tests/
├── zexc.toml # shared/static/CLI build config
├── Makefile
├── ROADMAP.md
└── README.md
```
---
## Documentation
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | System design, data path, module map |
| [docs/FORMAT.md](docs/FORMAT.md) | Binary `.zex` specification |
| [docs/API.md](docs/API.md) | Library API guide |
| [docs/STREAMING.md](docs/STREAMING.md) | Streaming encoder/decoder |
| [docs/BENCHMARKS.md](docs/BENCHMARKS.md) | How to measure vs zstd/lz4 |
| [docs/PACKAGING.md](docs/PACKAGING.md) | Install, features, release tarballs |
| [docs/zex.1](docs/zex.1) | Manual page |
| [ROADMAP.md](ROADMAP.md) | Near- and long-term plan |
---
## Building, testing, and benchmarks
```bash
# Unit, integration, and doctests
cargo test
# or: make test
# Optimized binary
make release
# Criterion microbenchmarks (ZEX only)
cargo bench --bench compression
# ZEX vs external zstd CLI (if installed)
cargo bench --bench vs_external
# Shell comparison table vs zstd (and lz4 if present)
make bench-compare
./scripts/bench_compare.sh /path/to/corpus.bin
# Package
make install-local
make release-tarball
```
---
## Performance expectations
ZEX is under active development. Mature C implementations (LZ4, Zstd) benefit from years of tuning; absolute “always faster and smaller than both” claims are not the project goal.
**Design targets:**
| Turbo | LZ4-class throughput orientation |
| Balanced | Practical default for mixed data |
| Max | Spend CPU for higher ratio |
| Secure / AEAD | Integrity and confidentiality with acceptable overhead |
Always benchmark on **your** data and hardware:
```bash
make bench-compare
```
---
## Roadmap
Planned work includes fuzzing and golden corpora, optimal parsing at high levels, turbo SIMD specialization, COVER-style dictionary training, seekable frames, and a stable C ABI. Details and sprint ordering live in **[ROADMAP.md](ROADMAP.md)**.
---
## Contributing
By contributing, you agree that your contributions are licensed under the same
**GPL-3.0-only** terms as the rest of the project (no dual-license unless agreed in writing).
1. Keep the codec **native** (no silent dependency on libzstd/liblz4 for the default path).
2. Prefer **additive** wire flags; do not break existing `ZEX1` decoders without a version bump.
3. Add tests for format and round-trip behavior.
4. Update `docs/FORMAT.md` when the on-wire layout changes.
5. Measure non-trivial performance claims with the provided bench scripts.
---
## License
**ZEX / zexc** is free software licensed under the
**[GNU General Public License version 3 only](LICENSE)**
(SPDX: **`GPL-3.0-only`**).
| **License** | GNU General Public License **v3 only** |
| **SPDX** | `GPL-3.0-only` |
| **Full text** | [LICENSE](LICENSE) |
| **“or later”?** | **No** — version 3 **only** (not GPL-3.0-or-later) |
```
Copyright (C) 2026 Ali Zain <alizain@zainiumdynamics.tech>
Website: https://zainiumdynamics.tech
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, version 3 of the License ONLY.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
```
### Author / contact
| **Name** | Ali Zain |
| **Email** | [alizain@zainiumdynamics.tech](mailto:alizain@zainiumdynamics.tech) |
| **Website** | [https://zainiumdynamics.tech](https://zainiumdynamics.tech) |
### What this means (short)
- You **may** use, study, modify, and redistribute ZEX under GPLv3.
- If you **distribute** binaries or derived works, you **must** provide corresponding source under the **same** GPLv3-only terms.
- Linking this library (including `libzexc` / the C API) into a larger program typically makes that combination a **covered work** under GPLv3 — proprietary closed-source distribution is **not** allowed without a separate commercial license (none is offered here by default).
- There is **no warranty**.
Design ideas were informed by the open specifications and literature around LZ4 and Zstandard; the implementation and `.zex` format are original.