Expand description
Pure Rust Zstandard codec.
zstandard implements the Zstandard compression
format without any C
dependency on the codec path. Frames produced by this crate decompress
cleanly with the official zstd library, and frames produced by zstd
decompress with this crate, for the format coverage implemented here.
The crate is intentionally narrow: the public API exposes one-shot and streaming encode/decode entry points, dictionary support, and a few frame-level helpers. Internal encoder behavior — parser strategies, hash tables, sequence statistics — is not part of the supported surface.
§One-shot
use zstandard::{decode_all, encode_all};
let compressed = encode_all(b"hello zstd")?;
assert_eq!(decode_all(&compressed)?, b"hello zstd");§Configurable encode
EncoderOptions controls compression level, block size, content
checksum, and the frame header fields. CompressionLevel accepts the
range -131072..=22: 1..=22 selects progressively more thorough parsers,
and the negative levels are Zstandard’s “fast mode”, trading ratio for
speed below what level 1 does.
use zstandard::{CompressionLevel, EncoderOptions, decode_all, encode_all_with_options};
let options = EncoderOptions {
compression_level: CompressionLevel::BETTER,
checksum: true,
..Default::default()
};
let compressed = encode_all_with_options(b"compress me", options)?;
assert_eq!(decode_all(&compressed)?, b"compress me");For finer control, ParameterOverrides replaces individual compression
parameters that the level would otherwise choose — window and table sizes,
search depth, minimum match, target length, and the parser Strategy.
Each parameter’s accepted range is published as a ParameterBounds
constant, and a value outside it is reported rather than clamped.
use zstandard::{EncoderOptions, ParameterOverrides, Strategy, decode_all, encode_all_with_options};
let options = EncoderOptions {
parameters: ParameterOverrides {
strategy: Some(Strategy::Lazy2),
window_log: Some(20),
..Default::default()
},
..Default::default()
};
let compressed = encode_all_with_options(b"a smaller window and a chosen parser", options)?;
assert_eq!(decode_all(&compressed)?, b"a smaller window and a chosen parser");§Dictionaries
Both raw-content and formatted Zstandard dictionaries are supported, and a
parsed dictionary can be reused across many calls via
EncoderDictionary and DecoderDictionary. They are separate types
because the two directions need different tables built from the same bytes
and neither should carry the other’s, which is the line upstream draws as
ZSTD_CDict and ZSTD_DDict; to use one dictionary both ways, build both
over the same Arc<[u8]>. train_dictionary builds one from sample data.
use zstandard::{decode_all_with_dict, encode_all_with_dict};
let dictionary = b"shared prefix dictionary";
let payload = b"shared prefix dictionary + payload";
let compressed = encode_all_with_dict(payload, dictionary)?;
assert_eq!(decode_all_with_dict(&compressed, dictionary)?, payload);§Streaming
StreamingEncoder and StreamingDecoder consume input in chunks and
produce output progressively. Both contexts can be reused for additional
frames via reset() after finish().
use zstandard::{DecoderOptions, EncoderOptions, StreamingDecoder, StreamingEncoder};
let input = b"streaming example data".repeat(1_000);
let mut encoder = StreamingEncoder::new(EncoderOptions::default())?;
let mut compressed = encoder.take_output();
for chunk in input.chunks(1_001) {
encoder.push(chunk)?;
compressed.extend_from_slice(&encoder.take_output());
}
encoder.finish()?;
compressed.extend_from_slice(&encoder.take_output());
let mut decoder = StreamingDecoder::new(DecoderOptions::default());
let mut restored = Vec::new();
for chunk in compressed.chunks(113) {
decoder.push(chunk)?;
restored.extend_from_slice(&decoder.take_output());
}
decoder.finish()?;
restored.extend_from_slice(&decoder.take_output());
assert_eq!(restored, input);§std::io adapters
io::Writer and io::Reader wrap the streaming contexts in the
std::io traits, so the codec composes with io::copy, BufReader, and
anything else that speaks Read/Write.
use std::io::{Read, Write};
use zstandard::io::{Reader, Writer};
let mut writer = Writer::new(Vec::new())?;
writer.write_all(b"through std::io")?;
let compressed = writer.finish()?;
let mut restored = Vec::new();
Reader::new(&compressed[..]).read_to_end(&mut restored)?;
assert_eq!(restored, b"through std::io");§Errors
All fallible operations return Result<T> (alias for
Result<T, Error>). Error reports malformed input, configuration
mistakes, and decoder limits that the input would exceed; the decoder is
the security boundary, so soundness on adversarial input is treated as a
bug — see SECURITY.md.
§Cargo features
internal-trace(maintainer-only, hidden) — enables block traces and stage-profiling APIs used by the project’s benchmark and parity tools. Not part of the supported surface.internal-fuzz(maintainer-only, hidden) — enables afuzzmodule exposing internal entry points consumed by the fuzz harness.asm-inspect(maintainer-only) — enables anasm_inspectmodule of#[no_mangle]shims used to inspect generated assembly.
§MSRV
Rust 1.96, as declared by rust-version in Cargo.toml and checked by CI.
The 2024 edition’s own floor is 1.85; this crate’s is higher.
Modules§
Structs§
- Block
Header - Parsed three-byte header that introduces every block inside a Zstandard frame.
- Compression
Level - Validated compression level,
-131072..=22(see Zstandard’sZSTD_compressionLevel). Higher values trade encode CPU for output size. - Decoder
- Reusable one-shot decoder that amortizes buffer allocation across calls.
- Decoder
Dictionary - A dictionary parsed for decompression and reusable across many decodes.
- Decoder
Options - Decoder configuration.
- Dictionary
Training Parameters - Parameters for
train_dictionary_with_parameters, mirroring upstream’sZDICT_fastCover_params_t. - Encoder
- Reusable one-shot encoder that amortizes buffer allocation across calls.
- Encoder
Dictionary - A dictionary parsed for compression and reusable across many encodes.
- Encoder
Options - Encoder configuration. Use
EncoderOptions::defaultfor sensible defaults, or override individual fields. A common pattern isEncoderOptions { compression_level: CompressionLevel::BETTER, ..Default::default() }. - Parameter
Bounds - Inclusive bounds for one compression parameter, upstream’s
ZSTD_bounds. - Parameter
Overrides - Compression parameters to use instead of the ones the level would choose,
upstream’s
ZSTD_c_windowLogand friends. - Skippable
Frame - Parsed eight-byte header of a skippable frame.
- Streaming
Decoder - Streaming decoder that consumes compressed input in chunks and produces decoded output progressively.
- Streaming
Encoder - Streaming encoder that consumes input in chunks and produces compressed output progressively.
- Trained
Dictionary - A trained dictionary and the parameters that produced it.
- Zstandard
Frame Header - Parsed contents of a Zstandard frame header.
Enums§
- Block
Type - Encoding type of a Zstandard block payload.
- Error
- All errors reported by encoder, decoder, and dictionary paths.
- Format
- Frame envelope, upstream’s
ZSTD_c_format/ZSTD_d_format. - Frame
Header - Result of
parse_frame_header: either a Zstandard frame or a skippable frame. - LdmMode
- Whether long-distance matching runs, upstream’s
ZSTD_c_enableLongDistanceMatching. - Literal
Compression Mode - Whether the literals section is entropy-coded, upstream’s
ZSTD_c_literalCompressionMode. - RowMatch
Finder Mode - Whether the row-based match finder runs, upstream’s
ZSTD_c_useRowMatchFinder. - Strategy
- Match-finding strategy, upstream’s
ZSTD_strategy.
Constants§
- BLOCK_
SIZE_ MAX - Largest payload size, in bytes, that a single Zstandard block may carry.
- DICTIONARY_
SIZE_ MIN - Smallest dictionary a caller may ask for, matching
ZDICT_DICTSIZE_MIN. - ZSTD_
MAGIC_ NUMBER - Magic number that introduces every Zstandard frame (little-endian on the wire).
Functions§
- compress_
bound - Largest frame the encoder can produce from
src_lenbytes underoptions. - decode_
all - Decode every Zstandard frame contained in
srcand concatenate the results into a freshVec<u8>. Skippable frames are silently passed over. - decode_
all_ with_ dict - Decode
srcusingdictas a dictionary. The slice is parsed on every call; preferdecode_all_with_prepared_dictwhen reusing the same dictionary across many calls. - decode_
all_ with_ dict_ and_ options - Like
decode_all_with_dictbut uses caller-suppliedDecoderOptions. - decode_
all_ with_ options - Like
decode_allbut uses caller-suppliedDecoderOptions. - decode_
all_ with_ prepared_ dict - Decode
srcusing a pre-parsed dictionary. The dictionary id, if any, must match the one declared by every Zstandard frame insrc. - decode_
all_ with_ prepared_ dict_ and_ options - Like
decode_all_with_prepared_dictbut uses caller-suppliedDecoderOptions. - decode_
into_ slice - Decode
srcintodst, returning how many bytes were written. - decode_
into_ slice_ with_ options - Like
decode_into_slicebut uses caller-suppliedDecoderOptions. - decompress_
bound - Upper bound on what every frame in
srcdecodes to, upstream’sZSTD_decompressBound. - decompress_
bound_ with_ format decompress_bound, for input written informat.- decompressed_
size - Total decoded size every frame in
srcdeclares, upstream’sZSTD_findDecompressedSize. - decompressed_
size_ with_ format decompressed_size, for input written informat.- encode_
all - Encode
srcas a single Zstandard frame using defaultEncoderOptionsand return the compressed bytes. - encode_
all_ with_ dict - Encode
srcusingdictas a dictionary. The slice is parsed on every call; useencode_all_with_prepared_dictfor repeated use. - encode_
all_ with_ dict_ and_ options - Like
encode_all_with_dictbut uses caller-suppliedEncoderOptions. - encode_
all_ with_ options - Like
encode_allbut uses caller-suppliedEncoderOptions. - encode_
all_ with_ prepared_ dict - Encode
srcusing a pre-parsed dictionary. Prefer this overencode_all_with_dictwhen reusing the same dictionary across many calls. - encode_
all_ with_ prepared_ dict_ and_ options - Like
encode_all_with_prepared_dictbut uses caller-suppliedEncoderOptions. - encode_
into_ slice - Encode
srcinto a caller-owned slice, returning the frame’s length. - find_
frame_ compressed_ size - Size in bytes of the first frame in
src, upstream’sZSTD_findFrameCompressedSize. - find_
frame_ compressed_ size_ with_ format find_frame_compressed_size, for input written informat.- parse_
block_ header - Convenience wrapper for
BlockHeader::parse. - parse_
frame_ header - Identify and parse the frame header at the start of
src. ReturnsError::BadMagicwhen the leading magic is neither Zstandard nor a skippable variant. - parse_
frame_ header_ with_ format parse_frame_header, for input written informat.- train_
dictionary - Train a dictionary of at most
capacitybytes fromsamples. - train_
dictionary_ with_ parameters - Train a dictionary with explicit parameters, the equivalent of
ZDICT_optimizeTrainFromBuffer_fastCover. - write_
skippable_ frame - Encode a skippable frame carrying
payload.magic_variantselects one of the 16 skippable magic numbers and must lie in0..16; the output is a freshVec<u8>containing the eight-byte header followed by the payload.
Type Aliases§
- Result
- Convenience alias for
core::result::Result<T, Error>used throughout the crate.