Expand description
A pure Rust implementation of the Zstandard compression format.
§Decompression
The decoding module contains the code for decompression.
Decompression can be achieved by using the decoding::StreamingDecoder
or the more low-level decoding::FrameDecoder
§Compression
The encoding module contains the code for compression.
Compression can be achieved by using the encoding::compress/encoding::compress_to_vec
functions or encoding::FrameCompressor
§structured-zstd
Pure Rust zstd implementation — managed fork of ruzstd.
§Benchmarks Dashboard
Historical benchmark charts are published to GitHub Pages:
- Performance dashboard
- Latest relative payload
- Latest benchmark delta report
- Latest full benchmark report
Note: the root Pages URL can be empty; benchmark charts live under /dev/bench/.
§Managed Fork
This is a maintained fork of KillingSpark/zstd-rs (ruzstd) by Structured World Foundation. We maintain additional features and hardening for the CoordiNode database engine.
Fork goals:
- Dictionary compression improvements (critical for per-label trained dictionaries in LSM-tree)
- Performance parity with C zstd for decompression (currently 1.4-3.5x slower)
- Full numeric compression levels (0 = default, 1–22 plus negative ultra-fast, with C zstd-compatible level numbering/API; not exact strategy/ratio parity at every level)
- No FFI — pure
cargo build, no cmake/system libraries (ADR-013 compliance)
Upstream relationship: We periodically sync with upstream but maintain an independent development trajectory focused on CoordiNode requirements.
§What is this
A pure Rust implementation of the Zstandard compression format, as defined in RFC 8878.
This crate contains a fully operational decompressor and a compressor that is usable but does not yet match the speed, ratio, or configurability of the original C library.
§Current Status
§Decompression
Complete RFC 8878 implementation. Performance: ~1.4-3.5x slower than C zstd depending on data compressibility.
§Compression
- Uncompressed blocks
- Fastest (roughly level 1)
- Default (roughly level 3)
- Better (roughly level 7)
- Best (roughly level 11)
-
Numeric levels
0(default),1–22, and negative ultra-fast levels viaCompressionLevel::from_level(n)(C zstd-compatible numbering) - Checksums
-
Frame Content Size —
FrameCompressorwrites FCS automatically;StreamingEncoderrequiresset_pledged_content_size()before first write - Dictionary compression
-
Streaming encoder (
io::Write)
§Compression Strategy Coverage
Implemented strategy/back-end coverage:
- Level 1: simple matcher (
Simple) - Levels 2-3:
Dfast - Level 4: row matcher (
Row) - Levels 5-22: hash-chain matcher (
HashChain) with lazy/lazy2 style tuning
Not yet implemented as dedicated strategy families:
greedybtoptbtultra
Current behavior for these missing families:
- numeric levels that require them are mapped to the closest implemented matcher configuration
§Dictionary Generation
When the dict_builder feature is enabled, the dictionary module can:
- build raw dictionaries with COVER (
create_raw_dict_from_source) - build raw dictionaries with FastCOVER (
create_fastcover_raw_dict_from_source) - finalize raw content into full zstd dictionary format (
finalize_raw_dict) - train+finalize in one pure-Rust flow (
create_fastcover_dict_from_source) - propagate I/O failures from dictionary-building APIs via
io::Resultreturn values
§Benchmarking
Performance tracking lives in BENCHMARKS.md. The suite compares structured-zstd against the C reference across small payloads, entropy extremes, a 100 MiB large-stream scenario, repository corpus fixtures, and optional local Silesia corpora. CI benchmark runs are now published as a multi-target matrix (x86_64-gnu, i686-gnu, x86_64-musl) and expose a relative-first payload (benchmark-relative.json) for dashboard filtering by target/stage/scenario/level/source.
Benchmark report files are generated by .github/scripts/run-benchmarks.sh and are kept as ignored local/CI artifacts rather than tracked files in this repository.
§Usage
§Compression
use structured_zstd::encoding::{compress, compress_to_vec, CompressionLevel};
let data: &[u8] = b"hello world";
// Named level
let compressed = compress_to_vec(data, CompressionLevel::Fastest);
// Numeric level (C zstd compatible: 0 = default, 1-22, negative for ultra-fast)
let compressed = compress_to_vec(data, CompressionLevel::from_level(7));use structured_zstd::encoding::{CompressionLevel, StreamingEncoder};
use std::io::Write;
let mut out = Vec::new();
let mut encoder = StreamingEncoder::new(&mut out, CompressionLevel::Fastest);
encoder.write_all(b"hello ")?;
encoder.write_all(b"world")?;
encoder.finish()?;§Decompression
use structured_zstd::decoding::StreamingDecoder;
use structured_zstd::io::Read;
let compressed_data: Vec<u8> = vec![];
let mut source: &[u8] = &compressed_data;
let mut decoder = StreamingDecoder::new(&mut source).unwrap();
let mut result = Vec::new();
decoder.read_to_end(&mut result).unwrap();§Dictionary-backed Decompression API
use structured_zstd::decoding::{DictionaryHandle, FrameDecoder, StreamingDecoder};
use structured_zstd::io::Read;
let compressed: Vec<u8> = vec![];
let dict_bytes: Vec<u8> = vec![];
let mut output = vec![0u8; 1024];
// Parse dictionary once, then reuse handle.
let handle = DictionaryHandle::decode_dict(&dict_bytes).unwrap();
let mut decoder = FrameDecoder::new();
let _written = decoder
.decode_all_with_dict_handle(compressed.as_slice(), &mut output, &handle)
.unwrap();
// Compatibility path: pass raw dictionary bytes directly.
let mut decoder = FrameDecoder::new();
let _written = decoder
.decode_all_with_dict_bytes(compressed.as_slice(), &mut output, &dict_bytes)
.unwrap();
// Streaming helpers exist for both handle- and bytes-based paths.
let mut source: &[u8] = &compressed;
let mut stream = StreamingDecoder::new_with_dictionary_handle(&mut source, &handle).unwrap();
let mut sink = Vec::new();
stream.read_to_end(&mut sink).unwrap();§Support the Project
USDT (TRC-20): TFDsezHa1cBkoeZT5q2T49Wp66K8t2DmdA
§License
Apache License 2.0
Contributions will be published under the same Apache 2.0 license.
Re-exports§
pub use io_std as io;