zexc 1.2.1

zex native lossless compressor — CLI zexc, library zexc, files *.zex
Documentation

ZEXC

zexc is a modern, pure-Rust lossless compression library and command-line tool. It implements an original codec and container format (.zex, magic ZEX1)—not a wrapper around LZ4 or Zstandard.

Author Ali Zain <alizain@zainiumdynamics.tech>
Website zainiumdynamics.tech
Language Rust 2024 (MSRV 1.85)
License GPL-3.0-only — GNU GPLv3 strict (no “or later”)
crates.io / crate zexc (use zexc::…)
CLI binary zexc
C library libzexc / zexc.h
File extension .zex
Version 1.2.0 (see crates.io/crates/zexc)

Highlights

  • Native algorithm — hash-chain match finding, lazy matching, dual hashing, optional Huffman literals, and rANS sequence coding with repeat offsets
  • Performance profiles — turbo (speed), balanced (default), max (ratio), levels 1–22
  • SIMD-oriented hot paths — wide match_len and fast RLE-aware copies
  • Streaming I/O — StreamEncoder / StreamDecoder implementing std::io::Write / Read
  • Dictionary history — shared dictionary as virtual prefix for small-message workloads
  • Optional AEAD — ChaCha20-Poly1305 with BLAKE3 password-based key derivation
  • Safe defaults — CRC32 content checksum, independent blocks, hard size caps
  • Production CLI — compress, decompress, info, train-dict; stdin/stdout pipes via -

ZEX is designed for systems that need a self-contained compressor: auditable Rust, no libzstd / liblz4 dependency, and a clear path from library API to CLI and packaging.


Table of contents

  1. Installation
  2. Command-line usage
  3. Library usage
  4. Language bindings
  5. Compression profiles
  6. Security model
  7. Project layout
  8. Documentation
  9. Building, testing, and benchmarks
  10. Performance expectations
  11. Roadmap
  12. License

Build configuration (zexc.toml)

Shared vs static vs CLI is not hard-coded in the Makefile. Edit zexc.toml:

[library]
shared = true    # libzexc.so
static = true    # libzexc.a
rlib = true

[binary]
standalone = true   # CLI: zexc

[features]
simd = true
encryption = true
c-api = true
make configure   # apply zexc.toml
make release     # build
make show-config

See docs/CONFIG.md.

Installation

From source

git clone https://gitlab.com/alizain.arch/zexc.git
cd zexc
cargo build --release --features cli
# binary: target/release/zexc

Install the CLI

From crates.io (after publish):

cargo install zexc

From source:

cargo install --path . --locked
# or
make install

Feature flags

Feature Default Description
std yes Standard library support
simd yes Wide-load match helpers
encryption yes ChaCha20-Poly1305 + password KDF
cli yes Build the zexc binary
c-api yes C ABI (libzexc / zexc.h)
# Minimal build without encryption
cargo build --release --no-default-features --features std,simd

Command-line usage

Primary form (positional input and output):

zexc compress   input.txt   output.zex
zexc decompress output.zex  recovered.txt
zexc info       file.zex
zexc test       file.zex
zexc <COMMAND> ...

Commands:
  compress       input  output.zex
  decompress     input.zex  output
  info           file.zex
  test           file.zex
  train-dict     out.dict samples...
  archive        create|list|extract
  seek-compress  input  output.zexs
  bench          file

Options (common)

Flag Meaning
-l N Level 1–22
-m turbo|balanced|max Profile
-e PASS Encrypt / decrypt password
-D dict Dictionary file
-T N Threads
-q Quiet
--rm Delete input after success

Pipes

cat data.bin | zexc compress - - > data.zex
zexc decompress data.zex - > data.bin

Dictionary

zexc train-dict http.dict samples/*.log
zexc compress msg.bin msg.zex -D http.dict
zexc decompress msg.zex msg.out -D http.dict

Archive

zexc archive create bundle.zexa a.txt b.bin
zexc archive list bundle.zexa
zexc archive extract bundle.zexa ./out

Library usage

Add the package to your Cargo.toml. crates.io name and import are both zexc.

[dependencies]
zexc = "1.1"

One-shot API

use zexc::{compress, decompress, CompressOptions, CompressionMode};

fn main() -> zexc::Result<()> {
    let opts = CompressOptions::default()
        .mode(CompressionMode::Balanced)
        .level(3)
        .threads(true);

    let compressed = compress(b"example payload", &opts)?;
    let plain = decompress(&compressed)?;
    assert_eq!(plain, b"example payload");
    Ok(())
}

Streaming API

use std::io::{Read, Write};
use zexc::{CompressOptions, StreamDecoder, StreamEncoder};

fn compress_stream(input: &[u8]) -> zexc::Result<Vec<u8>> {
    let mut out = Vec::new();
    let mut enc = StreamEncoder::new(&mut out, CompressOptions::default());
    enc.write_all(input)?;
    enc.finish()?;
    Ok(out)
}

fn decompress_stream(zex_bytes: &[u8]) -> std::io::Result<Vec<u8>> {
    let mut dec = StreamDecoder::new(std::io::Cursor::new(zex_bytes));
    let mut plain = Vec::new();
    dec.read_to_end(&mut plain)?;
    Ok(plain)
}

See docs/STREAMING.md for lifecycle rules (always call finish() on encode). Streaming does not support whole-frame AEAD; use the one-shot API for encrypted frames.

Dictionary and encryption (library)

use zexc::{
    compress_with_dict, decompress_with_options,
    CompressOptions, DecompressOptions, Dictionary,
};

#[cfg(feature = "encryption")]
use zexc::EncryptionKey;

let dict = Dictionary::train_simple(&samples, 64 * 1024)?;
let opts = CompressOptions::default().level(6).dictionary(dict.clone());

#[cfg(feature = "encryption")]
let opts = opts.encrypt(EncryptionKey::Password("secret".into()));

let frame = compress_with_dict(message, &opts, dict.clone())?;

let plain = decompress_with_options(
    &frame,
    &DecompressOptions {
        dictionary: Some(dict),
        #[cfg(feature = "encryption")]
        encryption: Some(EncryptionKey::Password("secret".into())),
        ..Default::default()
    },
)?;

More examples: examples/simple_compress.rs, examples/stream_compress.rs, examples/dictionary_train.rs.


Language bindings

Language Path Build
Python (PyO3 / maturin) bindings/python pip install maturin && cd bindings/python && maturin develop --release
Go (cgo, over the C ABI) bindings/go cargo build --release --features c-api && cd bindings/go && go test ./...

Both wrap the same native codec — no libzstd/liblz4 anywhere in the call chain. See each directory's README for the full API and licensing notes (GPLv3 applies to distributed binaries/wheels linking libzexc, same as the C ABI).


Compression profiles

Profile CLI / API Intended use
Turbo -m turbo / CompressionMode::Turbo Latency-sensitive paths; shallow search
Balanced -m balanced (default) General-purpose default
Max -m max Prefer ratio; deeper search and entropy

Levels 1–22 scale hash table size, chain depth, lazy matching, dual-hash, literal Huffman, and sequence rANS. Exact parameters are documented in docs/ARCHITECTURE.md.


Security model

Layer Behavior
Memory safety Codec implemented in safe Rust (no unsafe in the match/copy hot path)
Integrity Optional but default CRC32 over plaintext
Confidentiality Optional ChaCha20-Poly1305 over the compressed body
DoS bounds SecurityPolicy caps on block size and total output
Match validation Offsets and lengths checked against history (including dictionary)

Encryption authenticates the compressed payload; a wrong password fails closed (authentication error). Dictionary id mismatches also fail closed.

Use .secure() / --secure for stricter absolute size limits.


Project layout

zexc/   (this tree may still be named zex-compressor on disk)
├── src/
│   ├── algorithms/zex_native.rs    # core compressor / decompressor
│   ├── core/                       # frame encode, decode, stream, dictionary
│   ├── format/                     # header and block wire format
│   ├── utils/                      # SIMD, Huffman, rANS/FSE, CRC, security
│   ├── crypto/                     # AEAD + KDF (feature-gated)
│   ├── cli/                        # clap definitions
│   ├── lib.rs
│   └── main.rs                     # zexc binary
├── include/zexc.h                  # C API
├── bindings/
│   ├── python/                     # PyO3 / maturin
│   └── go/                         # cgo, over the C API
├── docs/                           # architecture, format, API, packaging
├── examples/
├── benches/
├── scripts/
├── fuzz/                           # cargo-fuzz targets + seed corpus
├── tests/
├── zexc.toml                       # shared/static/CLI build config
├── Makefile
├── ROADMAP.md
└── README.md

Documentation

Document Description
docs/ARCHITECTURE.md System design, data path, module map
docs/FORMAT.md Binary .zex specification
docs/API.md Library API guide
docs/STREAMING.md Streaming encoder/decoder
docs/BENCHMARKS.md How to measure vs zstd/lz4
docs/PACKAGING.md Install, features, release tarballs
docs/zex.1 Manual page
ROADMAP.md Near- and long-term plan

Building, testing, and benchmarks

# Unit, integration, and doctests
cargo test
# or: make test

# Optimized binary
make release

# Criterion microbenchmarks (ZEX only)
cargo bench --bench compression

# ZEX vs external zstd CLI (if installed)
cargo bench --bench vs_external

# Shell comparison table vs zstd (and lz4 if present)
make bench-compare
./scripts/bench_compare.sh /path/to/corpus.bin

# Package
make install-local
make release-tarball

Performance expectations

ZEX is under active development. Mature C implementations (LZ4, Zstd) benefit from years of tuning; absolute “always faster and smaller than both” claims are not the project goal.

Design targets:

Mode Goal
Turbo LZ4-class throughput orientation
Balanced Practical default for mixed data
Max Spend CPU for higher ratio
Secure / AEAD Integrity and confidentiality with acceptable overhead

Always benchmark on your data and hardware:

make bench-compare

Roadmap

Planned work includes fuzzing and golden corpora, optimal parsing at high levels, turbo SIMD specialization, COVER-style dictionary training, seekable frames, and a stable C ABI. Details and sprint ordering live in ROADMAP.md.


Contributing

By contributing, you agree that your contributions are licensed under the same
GPL-3.0-only terms as the rest of the project (no dual-license unless agreed in writing).

  1. Keep the codec native (no silent dependency on libzstd/liblz4 for the default path).
  2. Prefer additive wire flags; do not break existing ZEX1 decoders without a version bump.
  3. Add tests for format and round-trip behavior.
  4. Update docs/FORMAT.md when the on-wire layout changes.
  5. Measure non-trivial performance claims with the provided bench scripts.

License

ZEX / zexc is free software licensed under the
GNU General Public License version 3 only
(SPDX: GPL-3.0-only).

License GNU General Public License v3 only
SPDX GPL-3.0-only
Full text LICENSE
“or later”? No — version 3 only (not GPL-3.0-or-later)
Copyright (C) 2026 Ali Zain <alizain@zainiumdynamics.tech>
Website: https://zainiumdynamics.tech

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, version 3 of the License ONLY.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.

Author / contact

Name Ali Zain
Email alizain@zainiumdynamics.tech
Website https://zainiumdynamics.tech

What this means (short)

  • You may use, study, modify, and redistribute ZEX under GPLv3.
  • If you distribute binaries or derived works, you must provide corresponding source under the same GPLv3-only terms.
  • Linking this library (including libzexc / the C API) into a larger program typically makes that combination a covered work under GPLv3 — proprietary closed-source distribution is not allowed without a separate commercial license (none is offered here by default).
  • There is no warranty.

Design ideas were informed by the open specifications and literature around LZ4 and Zstandard; the implementation and .zex format are original.