rusty_zstd

A ground-up, pure-Rust Zstandard (RFC 8878) compressor and decompressor. #![deny(unsafe_code)] everywhere but one audited SIMD island, zero dependencies, no C, no *-sys crate, no FFI. Every frame it emits decompresses in facebook/zstd v1.5.7 and every frame that emits decompresses here — dual-gated on the Silesia corpus every commit.

Part of Remade With Rust by Mata Network. Full README, benchmark boards and methodology: the repository.

Install

cargo add rusty_zstd

[dependencies]
rusty_zstd = "0.1"

# …or for embedded / wasm targets with no `std`:
rusty_zstd = { version = "0.1", default-features = false, features = ["alloc"] }

The minimum supported configuration is no_std + alloc — every entry point returns or fills a Vec, so alloc is required. MSRV is 1.85.

Quick start

use rusty_zstd::{compress, decompress, DEFAULT_CLEVEL};

let data = b"the quick brown fox jumps over the lazy dog ".repeat(64);

let packed = compress(&data, DEFAULT_CLEVEL)?;   // level 3, as libzstd defaults
assert!(packed.len() < data.len());
assert_eq!(decompress(&packed)?, data);          // lossless, always
# Ok::<(), rusty_zstd::Error>(())

Levels run −7…22 with all nine libzstd strategies behind them. compress_with takes a CompressOptions for the checksum flag, content size and dictionary ID; compress_with_advanced exposes the full AdvancedOptions — window log, strategy, LDM, workers, job size, overlap.

Streaming, for data that does not fit in memory. The pump is libzstd-shaped: hand it an input slice and an output buffer, and it reports what it consumed and produced.

use rusty_zstd::{compress_stream_out_size, Compressor, Flush};

let mut enc = Compressor::new(3)?;
let mut out = Vec::new();
let mut buf = vec![0u8; compress_stream_out_size()];

for chunk in [b"first chunk ".as_slice(), b"second chunk".as_slice()] {
    let mut fed = 0;
    while fed < chunk.len() {
        let st = enc.stream(&chunk[fed..], &mut buf, Flush::Continue)?;
        fed += st.input_consumed;
        out.extend_from_slice(&buf[..st.output_produced]);
    }
}
loop {
    let st = enc.stream(&[], &mut buf, Flush::End)?;
    out.extend_from_slice(&buf[..st.output_produced]);
    if st.done {
        break;
    }
}
assert_eq!(rusty_zstd::decompress(&out)?, b"first chunk second chunk");
# Ok::<(), rusty_zstd::Error>(())

Dictionaries, trained from your own samples — the win that matters for many small records:

use rusty_zstd::{compress_using_dict, decompress_using_dict, train, Dictionary, TrainOptions};

let samples: Vec<Vec<u8>> = (0..256)
    .map(|i| format!("{{\"event\":\"click\",\"id\":{i},\"session\":\"abc123\"}}").into_bytes())
    .collect();
let refs: Vec<&[u8]> = samples.iter().map(|s| s.as_slice()).collect();

let raw = train(&refs, TrainOptions::default())?;    // fastcover, d=8 steps=4
let dict = Dictionary::from_bytes(&raw)?;

let packed = compress_using_dict(&samples[0], &dict, 3)?;
assert_eq!(decompress_using_dict(&packed, &dict)?, samples[0]);
# Ok::<(), rusty_zstd::Error>(())

What is in here

Items marked #[doc(hidden)] are campaign instrumentation for the repository's own benchmark harness. They carry no semver promise and may be renamed or removed in any release.

Correctness

Gated against facebook/zstd v1.5.7 as an external process, in both directions, every commit: C compresses → this decompresses bit-exact, and this compresses → C's zstd -t and zstd -d accept it. The XXH64 checksum is gated against the published vectors, and every SIMD kernel against its scalar twin.

License

MIT OR Apache-2.0, at your option. No GPL/LGPL and no C anywhere in the dependency tree — CI-enforced with cargo-deny.