compcol

A collection of compression algorithms in pure Rust.

compcol puts every supported algorithm — RLE, deflate, zlib, gzip, LZMA, xz, Zstandard, Brotli, LZ4, Snappy, LZW, LZO, LZX, Quantum, plus decoders for RAR 1/2/3/5 — behind one uniform streaming trait, with each algorithm gated by its own Cargo feature so downstream crates only pay for what they pull in. A runtime by-name factory makes algorithms selectable from configuration or a CLI flag, and a compcol binary turns the library into a Unix-style filter.

Design principles

• Pure Rust. No bindgen, no FFI, no C dependencies. The crate has zero runtime dependencies — nothing in [dependencies].
• 100% safe. unsafe_code = "forbid" is set crate-wide; the library never opts out.
• no_std. The library is #![no_std]. alloc is used by everything except the bare-bones rle algorithm; algorithms that need large windows or work buffers pull in alloc automatically.
• Streaming. The caller owns both buffers; the codec preserves its state across calls. Works in a 1-byte-on-both-sides streaming loop.
• Per-algorithm features. default = ["alloc", "rle", "deflate", "zlib", "gzip", "factory"]. Everything else is opt-in.
• all meta-feature. features = ["all"] is a single name that enables every algorithm — useful for downstream crates and the CLI install command instead of a 20-item feature list.

Supported algorithms

The RAR encoders are permanently Unsupported per RARLAB's unRAR license terms (every clean-room RAR reader — libarchive, The Unarchiver, 7-Zip — ships decoder-only for the same reason).

Every other algorithm decodes real-world output from its reference toolchain and produces output that the same reference toolchain accepts. Some encoders (zstd, brotli) lag the reference's compression ratio because they skip features like FSE-compressed Huffman weight tables (zstd) or encoder-side static-dictionary lookups for non-English text (brotli); the wire format is always conformant.

Library usage

# Cargo.toml
[dependencies]
compcol = { version = "0.1", features = ["gzip", "factory"] }

The trait

use compcol::{Algorithm, Encoder, Decoder, Progress, Error};

pub struct Progress {
    pub consumed: usize,  // bytes read from input
    pub written:  usize,  // bytes written to output
    pub done:     bool,   // true once finish() has fully drained
}

pub trait Encoder {
    fn encode(&mut self, input: &[u8], output: &mut [u8]) -> Result<Progress, Error>;
    fn finish(&mut self, output: &mut [u8]) -> Result<Progress, Error>;
    fn reset(&mut self);
}

pub trait Decoder {
    fn decode(&mut self, input: &[u8], output: &mut [u8]) -> Result<Progress, Error>;
    fn finish(&mut self, output: &mut [u8]) -> Result<Progress, Error>;
    fn reset(&mut self);

    /// Advance the decompressed stream by up to `n` bytes without
    /// emitting them. Default impl reads-and-discards through a small
    /// scratch buffer; algorithms can override for cheaper skipping.
    fn skip(&mut self, input: &[u8], n: usize) -> Result<Progress, Error>;
}

pub trait Algorithm {
    const NAME: &'static str;
    type Encoder: Encoder;
    type Decoder: Decoder;
    fn encoder() -> Self::Encoder;
    fn decoder() -> Self::Decoder;
}

One-shot helpers (compcol::vec)

For callers that already have the whole payload in memory:

use compcol::gzip::Gzip;
use compcol::vec::{compress_to_vec, decompress_to_vec, compress_to_vec_with};

let plain = b"hello world hello world hello world";

let compressed = compress_to_vec::<Gzip>(plain)?;
let decoded    = decompress_to_vec::<Gzip>(&compressed)?;
assert_eq!(decoded, plain);

// With explicit config:
let small = compress_to_vec_with::<Gzip>(
    plain, compcol::gzip::EncoderConfig { level: 9 },
)?;
# Ok::<(), compcol::Error>(())

compress_to_vec_with / decompress_to_vec_with accept the algorithm's EncoderConfig / DecoderConfig for tuning (level, quality, etc.). Available under the alloc feature — no std required.

Streaming through std::io (compcol::io)

For files, sockets, or any Read/Write source. All four directions are covered; pick by which side you control and which direction the bytes flow.

use std::io::{Read, Write};
use compcol::{Algorithm, gzip::Gzip};
use compcol::io::{EncoderWriter, DecoderReader};

// Write plaintext, get a compressed file.
let file = std::fs::File::create("hello.txt.gz")?;
let mut w = EncoderWriter::new(file, Gzip::encoder());
w.write_all(b"hello, gzip\n")?;
let _file = w.finish()?;                  // returns the inner File

// Read a compressed file as if it were plain text.
let file = std::fs::File::open("hello.txt.gz")?;
let mut r = DecoderReader::new(file, Gzip::decoder());
let mut decoded = String::new();
r.read_to_string(&mut decoded)?;
# Ok::<(), std::io::Error>(())

EncoderReader (compressed source out of a plain reader) and DecoderWriter (plain output out of a compressed writer) round out the set. Writers call finish on Drop best-effort — call finish() explicitly to catch errors. Requires the std feature.

Driving the trait directly

use compcol::gzip::{Encoder, Decoder};
use compcol::{Encoder as _, Decoder as _, Status};

let input = b"hello world hello world hello world";

// Encode.
let mut enc = Encoder::new();
let mut buf = [0u8; 256];
let mut encoded = Vec::new();
let mut consumed = 0;
while consumed < input.len() {
    let (p, status) = enc.encode(&input[consumed..], &mut buf).unwrap();
    encoded.extend_from_slice(&buf[..p.written]);
    consumed += p.consumed;
    if matches!(status, Status::InputEmpty) { break; }
}
loop {
    let (p, status) = enc.finish(&mut buf).unwrap();
    encoded.extend_from_slice(&buf[..p.written]);
    if matches!(status, Status::StreamEnd) { break; }
}

// Decode.
let mut dec = Decoder::new();
let mut decoded = Vec::new();
let mut c2 = 0;
while c2 < encoded.len() {
    let (p, status) = dec.decode(&encoded[c2..], &mut buf).unwrap();
    decoded.extend_from_slice(&buf[..p.written]);
    c2 += p.consumed;
    if matches!(status, Status::StreamEnd | Status::InputEmpty) { break; }
}
loop {
    let (p, status) = dec.finish(&mut buf).unwrap();
    decoded.extend_from_slice(&buf[..p.written]);
    if matches!(status, Status::StreamEnd) { break; }
}
assert_eq!(decoded, input);

Runtime selection via the factory

use compcol::{factory, Encoder as _, Decoder as _};

let mut enc = factory::encoder_by_name("gzip")
    .expect("gzip not compiled in");

let mut out = [0u8; 1024];
let p = enc.encode(b"hello", &mut out).unwrap();
// ...

println!("available algorithms: {:?}", factory::names());

factory::extension(name) returns the conventional file extension for each algorithm (e.g. "gz" for gzip, "zst" for zstd).

Skipping decompressed bytes

Useful for tar-style archive browsing — read a header, skip past the file body, read the next header:

use compcol::gzip::Decoder;
use compcol::Decoder as _;

let mut dec = Decoder::new();
// Skip past the first 100 decompressed bytes…
let p = dec.skip(&compressed[..], 100).unwrap();
// …then decode the next 50:
let mut out = [0u8; 50];
let p = dec.decode(&compressed[p.consumed..], &mut out).unwrap();

The default skip implementation just reads-and-discards through a small scratch buffer, so it works for every algorithm. Individual decoders are free to override with a smarter implementation when the format allows it (e.g. fast-forwarding through stored deflate blocks without LZ77 expansion).

CLI usage

The compcol binary ships with the crate. Install with:

cargo install --path . --features all

…or pick a subset:

cargo install --path . --features "gzip,zstd,brotli,lz4,factory"

Usage: compcol -t ALGO [OPTIONS] [INPUT]

Required:
    -t, --type ALGO         Algorithm (use --list to see what's compiled in)

Mode:
    -d, --decompress        Decompress instead of compress

Output (mutually exclusive):
    -c, --stdout            Write to stdout, keep input file
    -o, --output PATH       Write to PATH
    (default, INPUT given)  Write to <INPUT>.<ext> on compress, or strip
                            <ext> on decompress; remove INPUT on success
    (default, no INPUT)     Read stdin, write stdout

Misc:
    -k, --keep              Keep input file even in in-place mode
    -f, --force             Overwrite an existing output file
    -L, --list              List available algorithms and exit
    -V, --version           Print version and exit
    -h, --help              Print this help and exit

Examples

# Pipe-style use (gzip via stdin → stdout)
cat README.md | compcol -t gzip > README.md.gz

# In-place compression (mirrors gzip(1) semantics: removes the original)
compcol -t gzip README.md            # → README.md.gz, removes README.md

# Keep the original
compcol -t gzip -k README.md         # → README.md.gz, keeps README.md

# Decompress
compcol -t gzip -d README.md.gz      # → README.md, removes README.md.gz

# Force overwrite of an existing output file
compcol -t gzip -f README.md

# Round-trip into a pager
compcol -t xz -d archive.xz -c | less

# Mix algorithms
compcol -t zstd payload.bin          # → payload.bin.zst
compcol -t brotli payload.bin        # → payload.bin.br

# List what's compiled in
compcol --list

Exit codes: 0 success, 1 runtime / I/O error, 2 usage / argument error.

Cargo feature topology

[features]
default = ["alloc", "rle", "deflate", "zlib", "gzip", "factory"]
# Meta-feature: pulls in every algorithm. Equivalent to `--all-features`.
all     = ["alloc", "factory",
           "rle", "deflate", "zlib", "gzip",
           "lzma", "xz",
           "zstd", "brotli", "lz4", "snappy", "lzw",
           "lzo", "lzx", "quantum", "lzfse", "adc",
           "rar1", "rar2", "rar3", "rar5"]
alloc   = []
std     = ["alloc"]            # std::io::{Read,Write} adapters in compcol::io
factory = ["alloc"]            # by-name lookup, returns Box<dyn …>
rle     = []                   # no_std clean (alloc not required)
deflate = ["alloc"]
zlib    = ["deflate"]
gzip    = ["deflate"]
lzma    = ["alloc"]
xz      = ["lzma"]
zstd    = ["alloc"]
brotli  = ["alloc"]
lz4     = ["alloc"]
snappy  = ["alloc"]
lzw     = ["alloc"]
lzo     = ["alloc"]
lzx     = ["alloc"]
quantum = ["alloc"]
lzfse   = ["alloc"]            # decoder-only, bvx2 returns Unsupported
adc     = ["alloc"]
rar1    = ["alloc"]
rar2    = ["alloc"]
rar3    = ["alloc"]
rar5    = ["alloc"]

A bare --no-default-features build produces a library with just the trait surface — useful for the most constrained embedded targets. Adding rle gives an algorithm that doesn't need alloc. Adding any other algorithm feature pulls in alloc and the codec.

The alloc feature also enables compcol::vec (one-shot compress_to_vec / decompress_to_vec helpers). The std feature adds compcol::io (the Read/Write adapters) plus From<Error> for std::io::Error so adapter code can use ? freely.

features = ["all"] enables every algorithm and is the most ergonomic choice when you don't know in advance which formats you'll see.

The compcol binary is gated on features = ["factory"] so a --no-default-features library build doesn't try to compile it.

Errors

compcol::Error is a single crate-wide enum so trait objects work without GATs:

pub enum Error {
    Corrupt,             // generic malformed input
    UnexpectedEnd,       // finish() called mid-stream
    OutputTooSmall,      // codec has a minimum atomic output size
    BadHeader,           // container header malformed
    InvalidBlockType,    // deflate BTYPE=3, etc.
    InvalidHuffmanTree,  // code lengths violate Kraft inequality
    InvalidDistance,     // LZ77 back-reference out of range
    ChecksumMismatch,    // Adler-32 / CRC-32 mismatch
    TrailerMismatch,     // gzip ISIZE doesn't match output length
    Unsupported,         // option / mode this build doesn't implement
}

Development

cargo build                                                      # builds lib + bin (default features)
cargo build --no-default-features                                # bare no_std lib
cargo build --no-default-features --features rle                 # narrowest alloc-free build
cargo build --no-default-features --features all                 # every algorithm, still no_std

cargo test --all-features                                        # full test suite
cargo clippy --all-features --all-targets -- -D warnings         # lint clean
cargo fmt --all --check                                          # format clean

The crate currently ships with ~566 tests across 23 test binaries, including round-trip tests for every algorithm with an encoder, cross-validation against system gzip / xz / zstd / brotli / compress / lz4 / python3 lzo / python3 lzma, and hand-crafted hex fixtures for every decoder-only format (RAR 2/3/5, Quantum, LZX).

A simple benchmark harness lives at examples/bench.rs. Run it with:

cargo run --release --features all --example bench

It measures each compiled-in algorithm's encoder/decoder throughput and compression ratio on a small fixed corpus and compares against the system reference when one is installed. A snapshot of the output is kept in BENCH.md.

License

MIT. © 2026 Karpeles Lab Inc.