Skip to main content

Crate zstandard

Crate zstandard 

Source
Expand description

Pure Rust Zstandard codec.

zstandard implements the Zstandard compression format without any C dependency on the codec path. Frames produced by this crate decompress cleanly with the official zstd library, and frames produced by zstd decompress with this crate, for the format coverage implemented here.

The crate is intentionally narrow: the public API exposes one-shot and streaming encode/decode entry points, dictionary support, and a few frame-level helpers. Internal encoder behavior — parser strategies, hash tables, sequence statistics — is not part of the supported surface.

§One-shot

use zstandard::{decode_all, encode_all};

let compressed = encode_all(b"hello zstd")?;
assert_eq!(decode_all(&compressed)?, b"hello zstd");

§Configurable encode

EncoderOptions controls compression level, block size, content checksum, and the frame header fields. CompressionLevel accepts the range -131072..=22: 1..=22 selects progressively more thorough parsers, and the negative levels are Zstandard’s “fast mode”, trading ratio for speed below what level 1 does.

use zstandard::{CompressionLevel, EncoderOptions, decode_all, encode_all_with_options};

let options = EncoderOptions {
    compression_level: CompressionLevel::BETTER,
    checksum: true,
    ..Default::default()
};
let compressed = encode_all_with_options(b"compress me", options)?;
assert_eq!(decode_all(&compressed)?, b"compress me");

For finer control, ParameterOverrides replaces individual compression parameters that the level would otherwise choose — window and table sizes, search depth, minimum match, target length, and the parser Strategy. Each parameter’s accepted range is published as a ParameterBounds constant, and a value outside it is reported rather than clamped.

use zstandard::{EncoderOptions, ParameterOverrides, Strategy, decode_all, encode_all_with_options};

let options = EncoderOptions {
    parameters: ParameterOverrides {
        strategy: Some(Strategy::Lazy2),
        window_log: Some(20),
        ..Default::default()
    },
    ..Default::default()
};
let compressed = encode_all_with_options(b"a smaller window and a chosen parser", options)?;
assert_eq!(decode_all(&compressed)?, b"a smaller window and a chosen parser");

§Dictionaries

Both raw-content and formatted Zstandard dictionaries are supported, and a parsed dictionary can be reused across many calls via EncoderDictionary and DecoderDictionary. They are separate types because the two directions need different tables built from the same bytes and neither should carry the other’s, which is the line upstream draws as ZSTD_CDict and ZSTD_DDict; to use one dictionary both ways, build both over the same Arc<[u8]>. train_dictionary builds one from sample data.

use zstandard::{decode_all_with_dict, encode_all_with_dict};

let dictionary = b"shared prefix dictionary";
let payload = b"shared prefix dictionary + payload";

let compressed = encode_all_with_dict(payload, dictionary)?;
assert_eq!(decode_all_with_dict(&compressed, dictionary)?, payload);

§Streaming

StreamingEncoder and StreamingDecoder consume input in chunks and produce output progressively. Both contexts can be reused for additional frames via reset() after finish().

use zstandard::{DecoderOptions, EncoderOptions, StreamingDecoder, StreamingEncoder};

let input = b"streaming example data".repeat(1_000);

let mut encoder = StreamingEncoder::new(EncoderOptions::default())?;
let mut compressed = encoder.take_output();
for chunk in input.chunks(1_001) {
    encoder.push(chunk)?;
    compressed.extend_from_slice(&encoder.take_output());
}
encoder.finish()?;
compressed.extend_from_slice(&encoder.take_output());

let mut decoder = StreamingDecoder::new(DecoderOptions::default());
let mut restored = Vec::new();
for chunk in compressed.chunks(113) {
    decoder.push(chunk)?;
    restored.extend_from_slice(&decoder.take_output());
}
decoder.finish()?;
restored.extend_from_slice(&decoder.take_output());
assert_eq!(restored, input);

§std::io adapters

io::Writer and io::Reader wrap the streaming contexts in the std::io traits, so the codec composes with io::copy, BufReader, and anything else that speaks Read/Write.

use std::io::{Read, Write};
use zstandard::io::{Reader, Writer};

let mut writer = Writer::new(Vec::new())?;
writer.write_all(b"through std::io")?;
let compressed = writer.finish()?;

let mut restored = Vec::new();
Reader::new(&compressed[..]).read_to_end(&mut restored)?;
assert_eq!(restored, b"through std::io");

§Errors

All fallible operations return Result<T> (alias for Result<T, Error>). Error reports malformed input, configuration mistakes, and decoder limits that the input would exceed; the decoder is the security boundary, so soundness on adversarial input is treated as a bug — see SECURITY.md.

§Cargo features

  • internal-trace (maintainer-only, hidden) — enables block traces and stage-profiling APIs used by the project’s benchmark and parity tools. Not part of the supported surface.
  • internal-fuzz (maintainer-only, hidden) — enables a fuzz module exposing internal entry points consumed by the fuzz harness.
  • asm-inspect (maintainer-only) — enables an asm_inspect module of #[no_mangle] shims used to inspect generated assembly.

§MSRV

Rust 1.96, as declared by rust-version in Cargo.toml and checked by CI. The 2024 edition’s own floor is 1.85; this crate’s is higher.

Modules§

io
std::io adapters.

Structs§

BlockHeader
Parsed three-byte header that introduces every block inside a Zstandard frame.
CompressionLevel
Validated compression level, -131072..=22 (see Zstandard’s ZSTD_compressionLevel). Higher values trade encode CPU for output size.
Decoder
Reusable one-shot decoder that amortizes buffer allocation across calls.
DecoderDictionary
A dictionary parsed for decompression and reusable across many decodes.
DecoderOptions
Decoder configuration.
DictionaryTrainingParameters
Parameters for train_dictionary_with_parameters, mirroring upstream’s ZDICT_fastCover_params_t.
Encoder
Reusable one-shot encoder that amortizes buffer allocation across calls.
EncoderDictionary
A dictionary parsed for compression and reusable across many encodes.
EncoderOptions
Encoder configuration. Use EncoderOptions::default for sensible defaults, or override individual fields. A common pattern is EncoderOptions { compression_level: CompressionLevel::BETTER, ..Default::default() }.
ParameterBounds
Inclusive bounds for one compression parameter, upstream’s ZSTD_bounds.
ParameterOverrides
Compression parameters to use instead of the ones the level would choose, upstream’s ZSTD_c_windowLog and friends.
SkippableFrame
Parsed eight-byte header of a skippable frame.
StreamingDecoder
Streaming decoder that consumes compressed input in chunks and produces decoded output progressively.
StreamingEncoder
Streaming encoder that consumes input in chunks and produces compressed output progressively.
TrainedDictionary
A trained dictionary and the parameters that produced it.
ZstandardFrameHeader
Parsed contents of a Zstandard frame header.

Enums§

BlockType
Encoding type of a Zstandard block payload.
Error
All errors reported by encoder, decoder, and dictionary paths.
Format
Frame envelope, upstream’s ZSTD_c_format / ZSTD_d_format.
FrameHeader
Result of parse_frame_header: either a Zstandard frame or a skippable frame.
LdmMode
Whether long-distance matching runs, upstream’s ZSTD_c_enableLongDistanceMatching.
LiteralCompressionMode
Whether the literals section is entropy-coded, upstream’s ZSTD_c_literalCompressionMode.
RowMatchFinderMode
Whether the row-based match finder runs, upstream’s ZSTD_c_useRowMatchFinder.
Strategy
Match-finding strategy, upstream’s ZSTD_strategy.

Constants§

BLOCK_SIZE_MAX
Largest payload size, in bytes, that a single Zstandard block may carry.
DICTIONARY_SIZE_MIN
Smallest dictionary a caller may ask for, matching ZDICT_DICTSIZE_MIN.
ZSTD_MAGIC_NUMBER
Magic number that introduces every Zstandard frame (little-endian on the wire).

Functions§

compress_bound
Largest frame the encoder can produce from src_len bytes under options.
decode_all
Decode every Zstandard frame contained in src and concatenate the results into a fresh Vec<u8>. Skippable frames are silently passed over.
decode_all_with_dict
Decode src using dict as a dictionary. The slice is parsed on every call; prefer decode_all_with_prepared_dict when reusing the same dictionary across many calls.
decode_all_with_dict_and_options
Like decode_all_with_dict but uses caller-supplied DecoderOptions.
decode_all_with_options
Like decode_all but uses caller-supplied DecoderOptions.
decode_all_with_prepared_dict
Decode src using a pre-parsed dictionary. The dictionary id, if any, must match the one declared by every Zstandard frame in src.
decode_all_with_prepared_dict_and_options
Like decode_all_with_prepared_dict but uses caller-supplied DecoderOptions.
decode_into_slice
Decode src into dst, returning how many bytes were written.
decode_into_slice_with_options
Like decode_into_slice but uses caller-supplied DecoderOptions.
decompress_bound
Upper bound on what every frame in src decodes to, upstream’s ZSTD_decompressBound.
decompress_bound_with_format
decompress_bound, for input written in format.
decompressed_size
Total decoded size every frame in src declares, upstream’s ZSTD_findDecompressedSize.
decompressed_size_with_format
decompressed_size, for input written in format.
encode_all
Encode src as a single Zstandard frame using default EncoderOptions and return the compressed bytes.
encode_all_with_dict
Encode src using dict as a dictionary. The slice is parsed on every call; use encode_all_with_prepared_dict for repeated use.
encode_all_with_dict_and_options
Like encode_all_with_dict but uses caller-supplied EncoderOptions.
encode_all_with_options
Like encode_all but uses caller-supplied EncoderOptions.
encode_all_with_prepared_dict
Encode src using a pre-parsed dictionary. Prefer this over encode_all_with_dict when reusing the same dictionary across many calls.
encode_all_with_prepared_dict_and_options
Like encode_all_with_prepared_dict but uses caller-supplied EncoderOptions.
encode_into_slice
Encode src into a caller-owned slice, returning the frame’s length.
find_frame_compressed_size
Size in bytes of the first frame in src, upstream’s ZSTD_findFrameCompressedSize.
find_frame_compressed_size_with_format
find_frame_compressed_size, for input written in format.
parse_block_header
Convenience wrapper for BlockHeader::parse.
parse_frame_header
Identify and parse the frame header at the start of src. Returns Error::BadMagic when the leading magic is neither Zstandard nor a skippable variant.
parse_frame_header_with_format
parse_frame_header, for input written in format.
train_dictionary
Train a dictionary of at most capacity bytes from samples.
train_dictionary_with_parameters
Train a dictionary with explicit parameters, the equivalent of ZDICT_optimizeTrainFromBuffer_fastCover.
write_skippable_frame
Encode a skippable frame carrying payload. magic_variant selects one of the 16 skippable magic numbers and must lie in 0..16; the output is a fresh Vec<u8> containing the eight-byte header followed by the payload.

Type Aliases§

Result
Convenience alias for core::result::Result<T, Error> used throughout the crate.