zstd-rs

A pure-Rust implementation of Zstandard (RFC 8878). It includes both the compressor and the decompressor.

• Runs in the browser. The core is #![no_std] + alloc and #![forbid(unsafe_code)], with no C and no std. It builds for wasm32-unknown-unknown, and the resulting module has zero imports.

• Interoperates with the C library in both directions. Our frames decode with the reference zstd, and its frames decode with ours. This is tested at every level, from 0 bytes to 3 MB, with and without dictionaries. The official golden vectors also pass.

• Has real compression levels. It implements the reference's level table from -7 to 22:

  • the fast and dfast strategies;
  • greedy, lazy and lazy2 on hash chains;
  • an optimal parser (btopt/btultra/btultra2-style price DP).

  Compression ratio is within about 1% of C at the same level, and sometimes better on small inputs.

• Magicless frames. The reference's ZSTD_f_zstd1_magicless format is supported, which saves 4 bytes per frame.

• Supports dictionaries. It loads zstd-format dictionaries (zstd --train / ZDICT output, with entropy tables and repeat offsets) as well as raw-content dictionaries.

  • Dictionary match tables are built once and attached to each frame, never copied. This is what makes small messages fast.

• Safe to decode untrusted data. Malformed input returns an error and never panics, and every decode call takes a caller-supplied output cap.

• Parallel without owning threads. The core only plans and runs independent jobs. You bring the Executor (threads, Web Workers, or sequential), and the output is byte-identical whichever one you use.

Crates

Usage

use zstd_rs::{CompressionConfig, Compressor, Decompressor};

let data = b"hello hello hello hello".repeat(100);

// Every setting is explicit; presets are named constants (FAST = 1, DEFAULT = 3, HIGH = 19).
let mut c = Compressor::new(CompressionConfig::DEFAULT)?;
let mut frame = Vec::new();
c.compress(&data, None, &mut frame)?;

// The decoder never writes more than the cap you give it.
let mut out = Vec::new();
Decompressor::new().decompress(&frame, None, 1 << 20, &mut out)?;
assert_eq!(out, data);
# Ok::<(), zstd_rs::Error>(())

To use a dictionary, load it once and share it across frames and threads:

use zstd_rs::*;
# let dict_bytes = b"...trained with zstd --train...".to_vec();
# let msg = b"payload";
let dict = Dictionary::new(&dict_bytes, DictionaryFormat::Detect)?;
let enc = EncoderDictionary::new(&dict, 3)?;          // bound to a level, like a CDict
let dec = DecoderDictionary::new(dict)?;
let mut c = Compressor::new(CompressionConfig::MESSAGE)?; // level 3, no dict ID in the header
let mut frame = Vec::new();
c.compress(msg, Some(&enc), &mut frame)?;
let mut out = Vec::new();
Decompressor::new().decompress(&frame, Some(&dec), 4096, &mut out)?;
# Ok::<(), zstd_rs::Error>(())

To compress large inputs in parallel, split one frame into jobs and supply an executor:

use zstd_rs::{par, CompressionConfig};
# let data = vec![0u8; 1 << 20];
let jobs = par::plan(data.len(), 256 * 1024, 64 * 1024)?; // job size, overlap
let mut frame = Vec::new();
par::compress_frame(&par::Sequential, &CompressionConfig::DEFAULT, &data, None, &jobs, &mut frame)?;
// With zstd-rs-exec: par::compress_frame(&ThreadPool::new(8)?, ...) gives identical bytes.
# Ok::<(), zstd_rs::Error>(())

Other parallel entry points:

• par::compress_frames / par::decompress_frames work on independent frames.
• In the browser, the same pure functions are exported to JS: zr_frame_begin, zr_compress_job, zr_frame_end. A Web Worker pool can run the jobs.

Interoperability and testing

• tests/c_to_ours.rs decodes frames from the reference C library:
  • levels -5..19, sizes 0 B to 3 MB, text, random, runs and mixed data;
  • streaming frames without a content size;
  • ZDICT-trained and raw dictionaries;
  • multiple frames and skippable frames.
• tests/ours_to_c.rs takes our frames at 17 levels and checks them with both the C bulk and streaming decoders, and with ours.
• tests/golden.rs runs the reference repository's golden-decompression, golden-decompression-errors and golden-compression vectors.
• tests/par.rs checks that frames built from jobs are valid, and identical across executors.
• Fuzzing (zstd-rs/fuzz, cargo-fuzz) has three targets:
  • decode: arbitrary bytes, with and without a dictionary, never panics and respects the cap.
  • roundtrip: every level ± dictionary; our decoder and C's must both reproduce the input.
  • diff_c: differential decoding against C.

Where we are stricter than the reference

The reference's one-shot path accepts some malformed input that its own streaming path, or RFC 8878, rejects. We reject it too:

• raw and RLE blocks larger than Block_Maximum_Size;
• Huffman streams that are not consumed exactly, or that lack an end marker. The reference's fast 4-stream loop checks only the output count.

Legacy (pre-v0.8) zstd formats are not supported.

Benchmarks

The benchmarks run on a real ILM traffic corpus (chat, control, markdown, WebSocket JSON, Yjs updates and snapshots, file chunks).

• Each cell is ratio / compress µs / decompress µs per frame. Ratio is the sum of compressed bytes over the sum of raw bytes.
• -dict means a 32 KB ZDICT dictionary trained on a disjoint training corpus. No dictionary ID is written, for either implementation.
• dec:X<-zr3 decodes zstd-rs level-3 frames with decoder X, which gives a like-for-like decoder comparison.
• Yjs rows use the current CBOR byte-string wire form. -legacy rows use the old Array.from integer arrays.

V8 (Node 22, wasm32-unknown-unknown)

zstd-C is absent here because it does not build for this target.

Native (Apple M-series arm64), compared with the C library (zstd 1.5.7) at the same level

Decoder speed on identical frames (the C library's level-3 output)

ruzstd 0.8 could not decode any dictionary frame in this corpus (UninitializedHuffmanTable), so it has no dictionary column.

Summary

• Against ruzstd (the existing pure-Rust crate, Fastest is its only level):
  • Our level 1 compresses better on every class, and 2.8-7x faster. For example, chat is 0.856 vs 1.036, which means ruzstd grows chat.
  • Our decoder is 1.6-3x faster on the same frames.
  • We also decode dictionary frames, which ruzstd cannot.
• Against zstd-C at the same level:
  • Ratio is equal to within ±1% at levels 1 and 3. With dictionaries it is better on Yjs (0.429 vs 0.452) and within 2% on chat.
  • Small-frame speed is close: chat with a dictionary takes 1.0 µs vs 0.7 µs to compress, and 0.50 vs 0.34 µs to decode.
  • Large frames are 1.5-2.3x slower to compress and about 1.6x slower to decode.
  • Level 19 ratio is within 1-3%.
• The wasm module (encoder + decoder, zstd-rs-wasm) is 164 KB, or 64 KB gzipped. It has 0 imports.

Limitations

• The API is one-shot: whole input in, whole frame out. There is no streaming Read/Write adapter yet. The job API covers large inputs.
• Levels 13-22 use hash chains rather than the reference's binary trees. Their ratio is within about 1-3% of C, and they are slower on large inputs.
• On large inputs, native speed is about 45-70% of the C library for compression and about 60% for decompression. Small frames with a dictionary are closer.
• There is no built-in dictionary trainer. Train with zstd --train (ZDICT); both formats are loaded.
• Legacy zstd formats (pre-v0.8) are not supported.

License

Licensed under either of Apache License 2.0 or MIT at your option. The test vectors in zstd-rs/tests/golden come from the reference repository (BSD-3-Clause / GPL-2.0).

Releasing

Bump version in zstd-rs/Cargo.toml and zstd-rs-exec/Cargo.toml (and the zstd-rs dependency in zstd-rs-exec), commit, then push a matching tag:

git tag v0.1.0 && git push origin v0.1.0

.github/workflows/release.yml runs the full CI as its gate, refuses a tag that does not match every crate's version, publishes zstd-rs then zstd-rs-exec to crates.io with the organisation's CARGO_REGISTRY_TOKEN (skipping any version already published), and creates the GitHub release. A manual run from the Actions tab is a dry run unless dry_run is set to false.