Crate rusty_zstd

Expand description

Pure-Rust Zstandard (RFC 8878) codec.

The crate README is the module documentation, so its examples are compiled and run by cargo test --doc – an install guide that cannot go stale.

§rusty_zstd

A ground-up, pure-Rust Zstandard (RFC 8878) compressor and decompressor. #![deny(unsafe_code)] everywhere but one audited SIMD island, zero dependencies, no C, no *-sys crate, no FFI. Every frame it emits decompresses in facebook/zstd v1.5.7 and every frame that emits decompresses here — dual-gated on the Silesia corpus every commit.

Part of Remade With Rust by Mata Network. Full README, benchmark boards and methodology: the repository.

§Install

cargo add rusty_zstd

[dependencies]
rusty_zstd = "0.1"

# …or for embedded / wasm targets with no `std`:
rusty_zstd = { version = "0.1", default-features = false, features = ["alloc"] }

The minimum supported configuration is no_std + alloc — every entry point returns or fills a Vec, so alloc is required. MSRV is 1.85.

Feature	Default	What it adds
`std`	✅	`std::io`-shaped streaming, multi-threading, the trainer, runtime ISA dispatch
`alloc`	✅	implied by `std`; the minimum supported configuration
`profile`		the in-process stage profiler and its counters (off = zero overhead)

§Quick start

use rusty_zstd::{compress, decompress, DEFAULT_CLEVEL};

let data = b"the quick brown fox jumps over the lazy dog ".repeat(64);

let packed = compress(&data, DEFAULT_CLEVEL)?;   // level 3, as libzstd defaults
assert!(packed.len() < data.len());
assert_eq!(decompress(&packed)?, data);          // lossless, always

Levels run −7…22 with all nine libzstd strategies behind them. compress_with takes a CompressOptions for the checksum flag, content size and dictionary ID; compress_with_advanced exposes the full AdvancedOptions — window log, strategy, LDM, workers, job size, overlap.

Streaming, for data that does not fit in memory. The pump is libzstd-shaped: hand it an input slice and an output buffer, and it reports what it consumed and produced.

use rusty_zstd::{compress_stream_out_size, Compressor, Flush};

let mut enc = Compressor::new(3)?;
let mut out = Vec::new();
let mut buf = vec![0u8; compress_stream_out_size()];

for chunk in [b"first chunk ".as_slice(), b"second chunk".as_slice()] {
    let mut fed = 0;
    while fed < chunk.len() {
        let st = enc.stream(&chunk[fed..], &mut buf, Flush::Continue)?;
        fed += st.input_consumed;
        out.extend_from_slice(&buf[..st.output_produced]);
    }
}
loop {
    let st = enc.stream(&[], &mut buf, Flush::End)?;
    out.extend_from_slice(&buf[..st.output_produced]);
    if st.done {
        break;
    }
}
assert_eq!(rusty_zstd::decompress(&out)?, b"first chunk second chunk");

Dictionaries, trained from your own samples — the win that matters for many small records:

use rusty_zstd::{compress_using_dict, decompress_using_dict, train, Dictionary, TrainOptions};

let samples: Vec<Vec<u8>> = (0..256)
    .map(|i| format!("{{\"event\":\"click\",\"id\":{i},\"session\":\"abc123\"}}").into_bytes())
    .collect();
let refs: Vec<&[u8]> = samples.iter().map(|s| s.as_slice()).collect();

let raw = train(&refs, TrainOptions::default())?;    // fastcover, d=8 steps=4
let dict = Dictionary::from_bytes(&raw)?;

let packed = compress_using_dict(&samples[0], &dict, 3)?;
assert_eq!(decompress_using_dict(&packed, &dict)?, samples[0]);

§What is in here

Area	Surface
One-shot	`compress`, `decompress`, `decompress_into`, `compress_bound`, `content_size`, `find_frame_compressed_size`
Options	`CompressOptions`, `AdvancedOptions`, `DecompressOptions`, `CompressionParameters`, `Strategy`
Streaming	`Compressor`, `Decompressor`, `Flush`, `StreamStatus`, and the four recommended-buffer-size helpers
Dictionaries	`Dictionary`, `compress_using_dict`, `decompress_using_dict`, `compress_using_prefix` (patch-from), `train` + `TrainOptions` / `TrainAlgo`
Long-range	`LdmParams`, `DEFAULT_LONG_WINDOW_LOG`
Seekable	`compress_seekable`, `decompress_frame_at`, `parse_seek_table`, `SeekTable`, `SeekEntry`
Multi-thread	`compress_mt`, `default_nb_workers`, `resolve_job_size`, `overlap_size`
Inspection	`get_frame_header`, `FrameHeader`, `FrameKind`, `inspect_frames`, `ListedFrame`
Checksum	`xxh64` — the frame content hash, usable on its own

Items marked #[doc(hidden)] are campaign instrumentation for the repository’s own benchmark harness. They carry no semver promise and may be renamed or removed in any release.

§Correctness

Gated against facebook/zstd v1.5.7 as an external process, in both directions, every commit: C compresses → this decompresses bit-exact, and this compresses → C’s zstd -t and zstd -d accept it. The XXH64 checksum is gated against the published vectors, and every SIMD kernel against its scalar twin.

§License

MIT OR Apache-2.0, at your option. No GPL/LGPL and no C anywhere in the dependency tree — CI-enforced with cargo-deny.

Structs§

AdvancedOptions: Extra compressor knobs: LDM (--long), --rsyncable, target cblock size, MT.
CompressOptions: Knobs for compress_with.
CompressionParameters: libzstd ZSTD_compressionParameters for a level and size hint.
Compressor: Reusable compressor (one frame at a time).
DecompressOptions: Decoder knobs for decompress_with.
Decompressor: Reusable decompressor. Multi-frame. Window-bounded history.
Dictionary: A zstd dictionary (raw bytes or trained with entropy tables).
FrameHeader: Parsed Zstandard frame header (not skippable).
InProcessBench: In-process compress + decompress at a compression level.
LdmParams: LDM knobs (ZSTD_c_ldm*). Zero means “pick C-like defaults from windowLog”.
ListedFrame: One frame in a concatenated stream (-l / inspect_frames).
LoopTiming: Wall timing of a looped closure (the shared timer).
SeekEntry: One seek-table row.
SeekTable: Parsed seek table (does not include the skippable header itself).
StreamStatus: Bytes moved by one Compressor::stream / Decompressor::stream call.
TrainOptions: Knobs for train.

Enums§

Error: Codec error. Stable kinds only.
Flush: ZSTD_EndDirective.
FrameKind: First frame at src: a Zstd frame or a skippable frame.
Strategy: Match finder (libzstd ZSTD_strategy, values 1..=9).
TrainAlgo: Trainer algorithm.

Constants§

BLOCKSIZE_MAX: RFC 8878 / libzstd ZSTD_BLOCKSIZE_MAX.
DEFAULT_CLEVEL: Default compression level (matches libzstd ZSTD_CLEVEL_DEFAULT).
DEFAULT_FRAME_SIZE: Default independent-frame size (--seekable).
DEFAULT_LONG_WINDOW_LOG: Default --long windowLog (libzstd ZSTD_WINDOWLOG_LIMIT_DEFAULT path: 27).
DEFAULT_MAX_DICT: Default --maxdict (110 KiB), matching libzstd.
DEFAULT_WINDOW_MAX: Default decoder window cap (CLI -M default): 128 MiB.
DICT_ID_PUBLIC_MAX: Public Dictionary_IDs are below 2^31.
DICT_ID_PUBLIC_MIN: Minimum public Dictionary_ID (RFC 8878 reserved below this).
JOB_SIZE_MIN: Transparent minimum job size (512 KiB), unless overlap is larger.
MAGIC: Zstandard frame magic (little-endian 0xFD2FB528).
MAGIC_DICTIONARY: Trained dictionary magic (little-endian 0xEC30A437).
MAGIC_SKIPPABLE_MAX: Last skippable magic (0x184D2A5F).
MAGIC_SKIPPABLE_MIN: First skippable magic (0x184D2A50).
MAX_CLEVEL: Maximum compression level (matches libzstd --ultra 22).
MIN_CLEVEL: Minimum negative compression level (matches libzstd ZSTD_minCLevel).
NB_WORKERS_MAX: libzstd ZSTDMT_NBWORKERS_MAX on 64-bit.
SEEKABLE_MAGIC: Seek table footer magic (0x8F92EAB1).
SEEKABLE_SKIPPABLE_MAGIC: Skippable magic used by the seek table frame (0x184D2A5E).
VERSION: Library version (semver of this crate, not the zstd format).

Functions§

bench_roundtrip: Oneshot compress + decompress at level, timed as two separate phases. Checks decode(encode(x)) == x once, outside both timed regions.
bench_roundtrip_clocked: bench_roundtrip with an injected monotonic tick source.
compress: One-shot compress at level (-7..=22). Checksum on, content size in the header.
compress_bound: Worst-case compressed size for a single-pass frame (libzstd ZSTD_compressBound).
compress_mt: Compress src as concatenated independent frames, optionally in parallel.
compress_seekable: Compress src as independent frames plus a trailing seek table.
compress_seekable_adv: compress_seekable with LDM / rsyncable / target-cblock knobs.
compress_stream_in_size: Recommended input chunk (ZSTD_CStreamInSize).
compress_stream_out_size: Recommended output chunk (ZSTD_CStreamOutSize).
compress_using_dict: Compress src using a dictionary (raw or trained).
compress_using_dict_with: Compress with a dictionary and explicit checksum / Dictionary_ID knobs.
compress_using_prefix: Compress src with an external prefix (--patch-from / ZSTD_CCtx_refPrefix). No Dictionary_ID is written.
compress_with: One-shot compress with explicit options.
compress_with_advanced: compress_with_history plus LDM / rsyncable / target-cblock.
compress_with_history: One-shot compress with an optional dictionary or prefix (-D / --patch-from).
compress_with_params: One-shot compress with already-resolved compression parameters (--zstd=).
compression_params: Parameters C would pick at level for an optional size hint (None = unknown / large).
content_size: Frame_Content_Size of the first Zstd frame, skipping leading skippable frames. None means the size was not present in the header.
decompress: One-shot decompress of one or more concatenated frames (skippable frames ignored).
decompress_frame_at: Decompress the independent frame covering uncompressed offset (one frame).
decompress_into: Decompress into a caller-owned buffer, appending. Returns bytes written.
decompress_into_with: decompress_into with an explicit window cap.
decompress_stream_in_size: Recommended decompress input chunk (ZSTD_DStreamInSize).
decompress_stream_out_size: Recommended decompress output chunk (ZSTD_DStreamOutSize).
decompress_using_dict: One-shot decompress using a dictionary (ZSTD_decompress_usingDict).
decompress_using_dict_with: decompress_using_dict with an explicit window cap (-d --long).
decompress_using_prefix: One-shot decompress using a prefix (ZSTD_decompress_usingDDict / --patch-from).
decompress_using_prefix_with: decompress_using_prefix with an explicit window cap.
decompress_with: One-shot decompress with an explicit window cap.
default_nb_workers: std::thread::available_parallelism, clamped to 1..=NB_WORKERS_MAX (-T0).
default_overlap_log: Default --overlap-log when the caller passes 0 (zstd man: 6..=9 by strategy).
find_frame_compressed_size: Byte length of the first frame (Zstd or skippable), including checksum.
get_frame_header: Parse the first frame header (skippable or zstd). Does not consume blocks.
inspect_frames: Walk every concatenated frame (Zstd and skippable).
mbps: Uncompressed throughput in MB/s (src_len * loops / seconds) – the MEAN rate.
mbps_best: Uncompressed throughput in MB/s from the fastest single loop.
overlap_size: Overlap bytes reloaded from the previous job (overlapLog 1 = none, 9 = window).
parse_seek_table: Parse a seek table from the end of a seekable blob.
public_dict_id: Pick a public Dictionary_ID (RFC reserved ranges avoided unless forced).
resolve_job_size: Job size after C’s transparent minimum (max(512 KiB, overlap, 4*window if unset)).
time_loops: Run one at least once; keep running until min has elapsed (0 = once).
train: Train a zstd dictionary from samples. Output is a trained dict C can load.
xxh64: XXH64 with seed 0 – the only seed zstd uses for the content checksum.

Crate rusty_zstd

Crate rusty_zstd Copy item path

§rusty_zstd

§Install

§Quick start

§What is in here

§Correctness

§License

Structs§

Enums§

Constants§

Functions§

Crate rusty_zstd