zstd-complete

zstd-complete is a pure-Rust Zstandard compressor and decompressor. It supports bounded streaming compression, all standard positive compression levels, dictionaries, checksums, portable no_std operation, and safe public interfaces.

“Complete” means that the crate provides both sides of the Zstandard format. It does not mean that it reproduces the zstd C API or every advanced feature of the C library. The exact differences are listed below.

The project is a fork of ruzstd. Its high-performance compressor was derived from and translated from zstd C 1.5.7, then substantially reworked around Rust ownership, errors, and streaming. Format interoperability is a requirement; byte-for-byte identity with C output is not.

Install

[dependencies]
zstd-complete = "0.1"

The package name contains a hyphen, so Rust code imports it as zstd_complete.

The default features are std and hash. Disable default features for the lower-level no_std compressor and decompressor. Enable dict_builder to train raw-content dictionaries.

Bounded streaming compression

Encoder<W> implements std::io::Write and reports configuration, memory budget, and underlying I/O failures. It defaults to level 3, 8 MiB input frames, and a conservative 96 MiB working-memory budget.

# #[cfg(feature = "std")]
# fn main() -> Result<(), Box<dyn std::error::Error>> {
use std::{fs::File, io::Write};
use zstd_complete::encoding::{CompressionLevel, Encoder, EncoderOptions};

let input = b"data can be supplied in any number of writes";
let output = File::create("archive.zst")?;
let options = EncoderOptions::new(CompressionLevel::DEFAULT)
    .with_checksum(true);
let mut encoder = Encoder::new(output, options)?;
encoder.write_all(input)?;
encoder.finish()?;
# Ok(())
# }
# #[cfg(not(feature = "std"))]
# fn main() {}

Each configured input chunk is emitted as an independent Zstandard frame. Concatenated frames are valid Zstandard and keep memory bounded regardless of total input size. Smaller chunks can reduce compression ratio because matches do not cross frame boundaries. flush() closes the current frame; call finish() to write the final frame and recover the wrapped writer.

For small inputs, encode streams from a reader to a writer and encode_all returns a Vec<u8>. The older compress, compress_to_vec, and FrameCompressor interfaces remain available, including in no_std, but the bounded, fallible API is preferred for new std applications.

Opt-in parallel compression

Enable the non-default multithreading feature to compress independent frames on multiple worker threads. ParallelEncoder implements Write, preserves frame order, bounds the number of in-flight frames, and applies the configured memory limit to the aggregate worker estimate. A worker count of one delegates directly to the existing Encoder; the default single-threaded implementation does not contain channels, locks, atomics, or worker checks in its compression path.

# #[cfg(all(feature = "std", feature = "multithreading"))]
# fn main() -> Result<(), Box<dyn std::error::Error>> {
use std::{io::Write, num::NonZeroUsize};
use zstd_complete::encoding::{CompressionLevel, EncoderOptions, ParallelEncoder};

let options = EncoderOptions::new(CompressionLevel::DEFAULT)
    .with_memory_limit(256 * 1024 * 1024);
let mut encoder =
    ParallelEncoder::new(Vec::new(), options, NonZeroUsize::new(4).unwrap())?;
encoder.write_all(b"input split into ordered independent frames")?;
let compressed = encoder.finish()?;
# let _ = compressed;
# Ok(())
# }
# #[cfg(not(all(feature = "std", feature = "multithreading")))]
# fn main() {}

Parallelism begins only when at least two complete input frames are available, so inputs no larger than one configured frame do not gain throughput. Smaller frame chunks expose more parallel work but may reduce compression ratio because matches never cross frame boundaries.

Decompression

StreamingDecoder implements the crate's Read interface for one frame:

# #[cfg(feature = "std")]
# fn main() -> Result<(), Box<dyn std::error::Error>> {
use std::{fs::File, io::Read};
use zstd_complete::decoding::StreamingDecoder;

let input = File::open("archive.zst")?;
let mut decoder = StreamingDecoder::new(input)?;
let mut decoded = Vec::new();
decoder.read_to_end(&mut decoded)?;
# Ok(())
# }
# #[cfg(not(feature = "std"))]
# fn main() {}

Use FrameDecoder when you need incremental buffer collection, dictionary selection, explicit block limits, concatenated-frame handling, or control over skippable frames. StreamingDecoder currently stops after one frame; this is important when consuming the multi-frame output of a bounded Encoder.

Coverage compared with zstd C 1.5.7

The compressor implements the nine zstd strategy families used by the positive level table: Fast, DFast, Greedy, Lazy, Lazy2, BTLazy2, BTOpt, BTUltra, and BTUltra2. The public API intentionally exposes validated CompressionLevel values rather than C-style numeric tuning controls.

Interoperability tests cover Rust-produced output decoded by zstd C, C-produced fixtures decoded by Rust, checksummed frames, raw blocks, concatenated frames, and raw and formatted dictionaries. The retained differential matrix has 1,095 normal, target-size, and prepared-dictionary rows across 73 real-world fixtures; all rows decode. This is strong regression coverage, not a claim that every possible Zstandard stream has been exhaustively tested.

The release gate also compressed 35 files selected from the official zstd 1.5.7 source tree at levels 1, 3, 5, 8, 16, 19, and 22. All 245 Rust archives decoded with zstd C and reproduced their inputs. Rust also reproduced zstd C's decoded output for the four official golden decompression frames, including empty, RLE-first, zero-sequence, and 128 KiB block cases.

Safety

The normal public API is safe Rust: callers do not need an unsafe block. Performance-critical internals do use unsafe for bounded unaligned loads, initialized-buffer construction, entropy emission, and the decoder ring buffer. Those modules are private, their call-site invariants are documented, and portable scalar implementations remain available.

For the 0.1 release candidate, the local gate passed 758 tests (5 diagnostic tests ignored), strict Clippy, rustdoc, no_std wasm checks, forced-scalar tests, packaged-source compilation, 353 focused AddressSanitizer tests, and focused Miri checks of the ring buffer, short writes, unaligned loads, and prepared-sequence entropy path. Bounded ASan/libFuzzer campaigns completed 2,644 decoder cases and 424 public-encoder cases; encoder cases round-tripped through both Rust and zstd C. CI also defines native Windows and Apple-Silicon jobs. These checks are evidence supporting memory safety; they are not a formal proof or a warranty.

The translated compressor has received extensive differential testing, interoperability testing, and repeated hardware-counter benchmarking. It has not received an independent human line-by-line source review. This is disclosed so adopters can apply their own assurance requirements.

Benchmarks against zstd C

The primary real-world benchmark is the standard 211,938,580-byte Silesia corpus: 12 individually compressed text, executable, source, database, XML, and medical-image files ranging from 5 MiB to 51 MiB. Each implementation compressed every file three times in each of three perf stat samples, with input resident in memory and an owned compressed buffer returned. Rust used the public bounded API and its default 8 MiB frames; C 1.5.7 used ZSTD_compress2() and one frame per file.

Negative size gaps favor Rust. The result shows near-identical aggregate compression ratio and a remaining C speed advantage, especially at level 1. Every Rust archive was accepted by zstd C, and Rust decoded and byte-verified all 12 C level-3 archives. Corpus provenance, hashes, exact methodology, cycles, the seven-level official-source matrix, and reproduction commands are in RELEASE_BENCHMARKS_2026-08-30.md.

The smaller generated-shape benchmark below exposes cases that a mixed corpus can hide, particularly incompressible and all-zero data.

The public bounded encoder was tested first. Each generated input is 16 MiB; Rust used its default two 8 MiB frames while the C one-shot API used one frame. Each program compressed the input three times, and each counter is the mean of three perf stat runs. Every Rust archive passed zstd -t with zstd C 1.5.7.

This is the relevant end-user comparison, including the bounded encoder's frame-management cost and compression-ratio tradeoff. The all-zero case makes the largest remaining low-entropy overhead especially clear. Different frame counts mean this is a product-level comparison, not isolated algorithm parity.

The following narrower benchmark compares the retained one-shot compressor core with C under matching framing. It is useful for CPU attribution across strategy levels, but should not be substituted for the public-API table above.

The table below is the final release snapshot from 2026-08-30 on an AMD Ryzen 7 3700X, Linux x86-64, rustc 1.94.1. Both implementations compressed the same 1,022,035-byte real-world fixture 20 times in one process; each counter is the mean of three perf stat runs. Rust was built with --release and one code-generation unit. The C comparison is zstd 1.5.7 through ZSTD_compress2.

Negative gaps favor Rust. Instruction counts are used because wall-clock cycles are more sensitive to frequency and scheduling. This focused fixture shows where the remaining low-level CPU gap lies, but it is not a general throughput claim. The broader corpus, exact commands, output-size comparisons, upstream ruzstd comparison, streaming ratio measurements, and known limitations are in C_PORT_PAUSE_REPORT_2026-08-29.md and RELEASE_READINESS.md.

For historical context, on the same level-1 fixture the pre-port upstream ruzstd 0.9.1 Fastest path produced 619,101 bytes/run and used about 128.69 million instructions/run. This release produces 571,525 bytes/run and uses about 27.34 million instructions/run. That comparison covers level 1 only; upstream did not contain this release's complete C level table.

Building for performance

Normal Cargo release builds are supported. The benchmarked configuration uses one code-generation unit in the final application:

[profile.release]
codegen-units = 1

Cargo profiles belong to the final application, so this repository's setting does not automatically apply to downstream users. The optimization is optional: the external-consumer benchmark also showed a large advantage over upstream under Cargo's default release profile.

Do not set target-cpu=native for binaries distributed to other machines. Optimized x86-64 kernels use runtime feature detection, while scalar paths cover x86-64 without BMI2, AArch64, Windows, Apple platforms, and wasm no_std.

Dictionary training

Enable dict_builder for the dictionary module's raw-content dictionary trainer. It does not generate formatted dictionaries containing entropy tables. On the historical github-users sample set, its output was within 0.2% of the official trainer's compression result. Existing formatted dictionaries can still be consumed by the encoder and decoder.

Licensing and provenance

This package is distributed under BSD-3-Clause, selecting the BSD option offered by the Zstandard reference implementation.

This project is a fork of KillingSpark/zstd-rs, created by Moritz Borcherding and developed by its contributors. Substantial inherited Rust remains in the decoder and supporting format, entropy, dictionary, and I/O code. Its MIT copyright and permission notice is preserved as required attribution in LICENSES/ruzstd-MIT.txt; it is not a second outbound license choice for this package. See THIRD_PARTY_NOTICES.md for credit, provenance, and the detailed compressor source map.

The implementation is not represented as clean-room work and is not intended to remain a faithful C port. Future maintenance will improve the Rust design and performance directly while preserving Zstandard format interoperability; recurring audits of future zstd C releases are not part of that model.