Struct FrameDecoder 

pub struct FrameDecoder { /* private fields */ }

Low level Zstandard decoder that can be used to decompress frames with fine control over when and how many bytes are decoded.

This decoder is able to decode frames only partially and gives control over how many bytes/blocks will be decoded at a time (so you don’t have to decode a 10GB file into memory all at once). It reads bytes as needed from a provided source and can be read from to collect partial results.

If you want to just read the whole frame with an io::Read without having to deal with manually calling FrameDecoder::decode_blocks you can use the provided crate::decoding::StreamingDecoder wich wraps this FrameDecoder.

Workflow is as follows:

use structured_zstd::decoding::BlockDecodingStrategy;

use std::io::{Read, Write};

// no_std environments can use the crate's own Read traits
use structured_zstd::io::{Read, Write};

fn decode_this(mut file: impl Read) {
    //Create a new decoder
    let mut frame_dec = structured_zstd::decoding::FrameDecoder::new();
    let mut result = Vec::new();

    // Use reset or init to make the decoder ready to decode the frame from the io::Read
    frame_dec.reset(&mut file).unwrap();

    // Loop until the frame has been decoded completely
    while !frame_dec.is_finished() {
        // decode (roughly) batch_size many bytes
        frame_dec.decode_blocks(&mut file, BlockDecodingStrategy::UptoBytes(1024)).unwrap();

        // read from the decoder to collect bytes from the internal buffer
        let bytes_read = frame_dec.read(result.as_mut_slice()).unwrap();

        // then do something with it
        do_something(&result[0..bytes_read]);
    }

    // handle the last chunk of data
    while frame_dec.can_collect() > 0 {
        let x = frame_dec.read(result.as_mut_slice()).unwrap();

        do_something(&result[0..x]);
    }
}

fn do_something(data: &[u8]) {
    std::io::stdout().write_all(data).unwrap();
}

Implementations

impl FrameDecoder

pub fn new() -> FrameDecoder

This will create a new decoder without allocating anything yet. init()/reset() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations

pub fn workspace_size(&self) -> usize

Heap bytes currently held by the decoder’s lazily-grown workspace: the decode-window buffer plus the per-block literal/content buffers and the entropy tables. Returns 0 before the first frame is initialised (no workspace allocated yet). The window allocation dominates and grows with the frame’s window size; this is the value to track for decode-time memory pressure, mirroring the workspace term of upstream ZSTD_sizeof_DCtx. Shared dictionaries (ref-counted handles) are not counted, matching upstream excluding refDDict memory.

pub fn set_content_checksum(&mut self, mode: ContentChecksum)

Select how the frame’s optional content checksum is handled (compute, expose, verify, or skip). See ContentChecksum. Default ContentChecksum::EmitOnly. Takes effect on the next decode; safe to call between frames on a reused decoder.

pub fn enable_per_block_checksums(&mut self)

Available on crate features hash and lsm only.

Opt in to per-block XXH64 verification during decode. Default off; zero cost when disabled. Each block’s decompressed bytes are XXH64-hashed (low 32 bits) and appended to Self::computed_block_checksums as the decode progresses. Callers compare the captured digests against externally-stored expected values (e.g. from a per-block sidecar in the containing application protocol).

Behind all(feature = "lsm", feature = "hash") — the XXH64 primitive lives behind the hash feature, so this method only compiles when both are enabled.

pub fn computed_block_checksums(&self) -> &[u32]

Available on crate features hash and lsm only.

Per-block XXH64 (low 32 bits) digests captured during the current frame’s decode. Empty unless Self::enable_per_block_checksums was called before Self::decode_all / Self::reset.

Reset at the start of every new frame.

Behind all(feature = "lsm", feature = "hash").

pub fn expect_dict_id(&mut self, expected: Option<u32>)

Available on crate feature lsm only.

Pin the expected Dictionary_ID for the next frame.

When expected is set, Self::init / Self::reset validate it against the parsed frame header BEFORE any block decode work runs. A mismatch returns crate::decoding::errors::FrameDecoderError::UnexpectedDictId before any block decode and before any output is produced. Scratch buffer allocation / reservation for the decode pipeline happens during frame-header parsing, which is already complete when this validation fires — the cost of scratch sizing is paid even on a mismatched header. The guarantee is “no block decode, no XXH64 init, no partial output”, not “zero allocation”.

Some(0) is treated as “no dictionary expected”: a frame whose header omits the optional Dictionary_ID field (flag value 0) passes the check; a frame that carries an explicit non-zero id fails.

None (default) disables the check.

Primary use case: post-AEAD-decrypt sanity check in wire-format consumers (e.g. lsm-tree’s encrypted block format pins the dict_id baked into the AAD against the inner zstd frame’s dict_id to defeat dict-substitution attacks).

NOT a replacement for AEAD authentication. NOT the same semantic as donor ZSTD_d_windowLogMax (which is a ceiling-style limit, separate concern).

pub fn expect_window_descriptor(&mut self, expected: Option<u8>)

Available on crate feature lsm only.

Pin the expected raw Window_Descriptor byte (RFC 8878 §3.1.1.1.2 layout: (exp << 3) | mantissa) for the next frame.

When expected is set, Self::init / Self::reset validate it against the parsed frame header BEFORE any block decode work runs. A mismatch returns crate::decoding::errors::FrameDecoderError::UnexpectedWindowDescriptor.

Single-segment frames omit the Window_Descriptor byte from the wire entirely. Setting an expectation while receiving a single-segment frame fails the check with found: None — there is no on-wire byte to match against, which is reported explicitly rather than silently passing.

None (default) disables the check.

Byte-exact equality, NOT a ceiling. Donor ZSTD_d_windowLogMax is a separate ceiling-style limit available through the C FFI surface; this method is for strict equality validation against a pinned expectation (e.g. lsm-tree’s wire format pins the window descriptor from the AAD to defeat decompression-bomb-swap attacks).

pub fn set_magicless(&mut self, magicless: bool)

Enable or disable magicless frame format (ZSTD_f_zstd1_magicless). When set to true, subsequent [init] / [reset] calls expect the frame header to begin directly with the frame-header descriptor — no 4-byte magic number prefix. Default false. Must match the encoder’s magicless setting; the format is unambiguous only when the caller knows it out-of-band.

Note: magicless mode also disables skippable-frame detection. The 0x184D2A50..=0x184D2A5F skippable-frame magic range is only recognised when the 4-byte magic prefix is consumed, so decode_all / init / reset will treat a skippable frame at the head of a magicless stream as a malformed frame header (bad descriptor / window-size error) instead of skipping it. Mixed-format streams that interleave skippable frames must be pre-split by the caller; set_magicless(true) is only safe when the entire stream is known to be magicless zstd frames.

pub fn init(&mut self, source: impl Read) -> Result<(), FrameDecoderError>

init() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations

Note that all bytes currently in the decodebuffer from any previous frame will be lost. Collect them with collect()/collect_to_writer()

equivalent to reset()

pub fn init_with_dict_handle( &mut self, source: impl Read, dict: &DictionaryHandle, ) -> Result<(), FrameDecoderError>

Initialize the decoder for a new frame using a pre-parsed dictionary handle.

If the frame header has a dictionary ID, this validates it against dict.id() and returns FrameDecoderError::DictIdMismatch on mismatch.

If the header omits the optional dictionary ID, this still applies the provided dictionary handle.

Warning

This method always applies dict unless the frame header contains a non-matching dictionary ID. Callers must only use this API when they already know the frame was encoded with the provided dictionary, even if the frame header omits the dictionary ID or encodes an explicit dictionary ID of 0.

Passing a dictionary for a frame that was not encoded with it can silently corrupt the decoded output.

pub fn reset(&mut self, source: impl Read) -> Result<(), FrameDecoderError>

reset() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations

Note that all bytes currently in the decodebuffer from any previous frame will be lost. Collect them with collect()/collect_to_writer()

equivalent to init()

pub fn reset_with_dict_handle( &mut self, source: impl Read, dict: &DictionaryHandle, ) -> Result<(), FrameDecoderError>

Reset this decoder for a new frame using a pre-parsed dictionary handle.

If the frame header has a dictionary ID, this validates it against dict.id() and returns FrameDecoderError::DictIdMismatch on mismatch.

If the header omits the optional dictionary ID, this still applies the provided dictionary handle.

Warning

This method always applies dict unless the frame header contains a non-matching dictionary ID. Callers must only use this API when they already know the frame was encoded with the provided dictionary, even if the frame header omits the dictionary ID or encodes an explicit dictionary ID of 0.

Passing a dictionary for a frame that was not encoded with it can silently corrupt the decoded output.

pub fn add_dict(&mut self, dict: Dictionary) -> Result<(), FrameDecoderError>

Add a dictionary that can be selected dynamically by frame dictionary ID.

Returns FrameDecoderError::DictAlreadyRegistered if the ID is already registered (either as owned or shared).

pub fn add_dict_from_bytes( &mut self, raw_dictionary: &[u8], ) -> Result<(), FrameDecoderError>

Parse and add a serialized dictionary blob.

pub fn add_dict_handle( &mut self, dict: DictionaryHandle, ) -> Result<(), FrameDecoderError>

Available on target_has_atomic=ptr only.

Add a pre-parsed dictionary handle for reuse across decoders.

This API is available on targets with pointer-width atomics (target_has_atomic = "ptr").

Returns FrameDecoderError::DictAlreadyRegistered if the ID is already registered (either as owned or shared).

pub fn force_dict(&mut self, dict_id: u32) -> Result<(), FrameDecoderError>

pub fn content_size(&self) -> u64

Returns how many bytes the frame contains after decompression

pub fn get_checksum_from_data(&self) -> Option<u32>

Returns the checksum that was read from the data. Only available after all bytes have been read. It is the last 4 bytes of a zstd-frame

pub fn get_calculated_checksum(&self) -> Option<u32>

Available on crate feature hash only.

Returns the checksum that was calculated while decoding. Only a sensible value after all decoded bytes have been collected/read from the FrameDecoder. Returns None when the frame header has content_checksum_flag = 0: no hash is computed for such frames (the post-decode XXH64 pass was a 63 % decode-wall hotspot on flag-off frames; skipping it when the frame format declares no trailing digest avoids that wasted work).

pub fn verify_content_checksum(&self) -> Result<(), FrameDecoderError>

Available on crate feature hash only.

Compare the frame’s stored content checksum against the digest the decoder computed, returning FrameDecoderError::ChecksumMismatch on disagreement. No-op unless the mode is ContentChecksum::Verify and the frame carries a trailing checksum.

decode_all and the streaming reader call this automatically. Callers driving decode_blocks directly invoke it themselves once per frame, after the frame is fully decoded AND fully drained (e.g. via collect), so both the stored value and the running digest are final.

pub fn bytes_read_from_source(&self) -> u64

Counter for how many bytes have been consumed while decoding the frame

pub fn is_finished(&self) -> bool

Whether the current frames last block has been decoded yet If this returns true you can call the drain* functions to get all content (the read() function will drain automatically if this returns true)

pub fn blocks_decoded(&self) -> usize

Counter for how many blocks have already been decoded

pub fn decode_blocks( &mut self, source: impl Read, strat: BlockDecodingStrategy, ) -> Result<bool, FrameDecoderError>

Decodes blocks from a reader. It requires that the framedecoder has been initialized first. The Strategy influences how many blocks will be decoded before the function returns This is important if you want to manage memory consumption carefully. If you don’t care about that you can just choose the strategy “All” and have all blocks of the frame decoded into the buffer

pub fn decode_blocks_partial( &mut self, source: impl Read, start_block: u32, end_block: u32, resume: Option<ResumeInput<'_>>, emit_resume: bool, ) -> Result<PartialDecode, FrameDecoderError>

Available on crate feature lsm only.

Decode the inner blocks [start_block, end_block) of the current frame and return their decompressed bytes as one contiguous buffer.

Serves two consumer needs with one call:

• Range-query performance: decode only the inner zstd blocks that cover a key range instead of the whole frame. Blocks before start_block are decoded into the window (zstd blocks share one window, so a leading block’s bytes may be the match source for an in-range block and cannot simply be skipped) but their output is not returned; blocks at or after end_block are not decoded at all, which is the trailing-block work saving. Map a decompressed byte offset to a block index with FrameEmitInfo::decompressed_byte_range.
• Best-effort recovery: if a block decode fails, decoding stops, the clean prefix of in-range output is preserved in PartialDecode::data, and the failure is reported via PartialDecode::stopped_at. Passing (0, u32::MAX) decodes the whole frame, stopping at the first corrupt block (pure recovery).

end_block is exclusive; pass u32::MAX to decode to the end of the frame. Call on a freshly reset decoder (it decodes from the frame’s first block).

Resume (cold incremental / top-up)

A plain call drains its in-range output from the match window on return, so two consecutive calls cannot resume one another and growing a decoded extent would mean re-decoding the covering prefix from block 0 (O(extent) per growth, O(N²) for a forward walk). The resume / emit_resume arguments make a symmetric one-call grow-loop possible:

• emit_resume = true captures the cross-block carry-over state (entropy tables + repcode history + the next block index / output offset) into PartialDecode::resume_state. The entropy-table snapshot clone is only paid when this is set. The snapshot is None when the decode reaches the frame’s last block (PartialDecode::frame_finished): there is no following block to resume from, so an incremental walk stops on frame_finished rather than on a None snapshot.
• resume = Some(ResumeInput) continues from a previously emitted ResumeState WITHOUT re-decompressing the preceding blocks: the match window is primed from ResumeInput::window_prime and the entropy/repcode tables are restored from the state, so a Repeat_Mode resume block resolves byte-identically to a contiguous decode — even across a dropped (cold) decoder.

When resume is Some, decoding resumes at ResumeState::block_index and the start_block argument is ignored (pass resume.state.block_index()); position source at that block’s compressed frame offset (FrameEmitInfo::blocks[block_index].offset_in_frame). After a resumed call, bytes_read_from_source and any stopped_at offsets are relative to the repositioned source.

Dictionaries: ResumeState does NOT carry the dictionary content. For a dictionary frame, attach the dictionary to the resuming decoder the same way as for a fresh decode — reset with the dictionary registered (or reset_with_dict_handle) BEFORE this call — so dict-sourced matches near the frame start resolve. The caller already holds the dictionary (it supplied it at encode time), so re-supplying it on resume is free; storing it in the snapshot would only duplicate it. The resume guard records the applied dictionary’s identity and rejects (FrameDecoderError::ResumeFrameMismatch) a resume whose active dictionary differs from the one the snapshot was captured under.

Errors

Returns FrameDecoderError::NotYetInitialized if the decoder has not been reset, FrameDecoderError::InvalidBlockRange if the effective start exceeds end_block, FrameDecoderError::ResumeWindowTooShort if resume’s window_prime is shorter than the match window the resume block can reach back into (min(window_size, output_offset)), and FrameDecoderError::ResumeFrameMismatch if the snapshot was captured from a frame with a different decode shape / dictionary, or (with the hash feature) a window_prime whose content does not match what was captured — all rejected up front rather than silently mis-resolving matches. A corrupt block is NOT an Err here: it is reported via PartialDecode::stopped_at so the clean prefix survives.

pub fn collect(&mut self) -> Option<Vec<u8>>

Collect bytes and retain window_size bytes while decoding is still going on. After decoding of the frame (is_finished() == true) has finished it will collect all remaining bytes

pub fn collect_to_writer(&mut self, w: impl Write) -> Result<usize, Error>

Collect bytes and retain window_size bytes while decoding is still going on. After decoding of the frame (is_finished() == true) has finished it will collect all remaining bytes

pub fn can_collect(&self) -> usize

How many bytes can currently be collected from the decodebuffer, while decoding is going on this will be lower than the actual decodbuffer size because window_size bytes need to be retained for decoding. After decoding of the frame (is_finished() == true) has finished it will report all remaining bytes

pub fn decode_from_to( &mut self, source: &[u8], target: &mut [u8], ) -> Result<(usize, usize), FrameDecoderError>

Decodes as many blocks as possible from the source slice and reads from the decodebuffer into the target slice The source slice may contain only parts of a frame but must contain at least one full block to make progress

By all means use decode_blocks if you have a io.Reader available. This is just for compatibility with other decompressors which try to serve an old-style c api

Returns (read, written), if read == 0 then the source did not contain a full block and further calls with the same input will not make any progress!

Note that no kind of block can be bigger than 128kb. So to be safe use at least 128*1024 (max block content size) + 3 (block_header size) + 18 (max frame_header size) bytes as your source buffer

You may call this function with an empty source after all bytes have been decoded. This is equivalent to just call decoder.read(&mut target)

pub fn decode_all( &mut self, input: &[u8], output: &mut [u8], ) -> Result<usize, FrameDecoderError>

Decode multiple frames into the output slice.

input must contain an exact number of frames. Skippable frames are allowed and will be skipped during decode.

output must be large enough to hold the decompressed data. If you don’t know how large the output will be, use FrameDecoder::decode_blocks instead.

This calls FrameDecoder::init, and all bytes currently in the decoder will be lost.

Returns the number of bytes written to output.

pub fn decode_all_with_skippable_visitor<F>( &mut self, input: &[u8], output: &mut [u8], visitor: F, ) -> Result<usize, FrameDecoderError>

where F: FnMut(u8, &[u8]),

Available on crate feature lsm only.

Decode multiple frames into the output slice, invoking visitor for every skippable frame encountered before advancing past it.

input must contain an exact number of frames. Skippable frames (RFC 8878 §3.1.2 magic numbers 0x184D2A50..=0x184D2A5F) are allowed and will be both visited AND skipped: the visitor gets (magic_variant, payload) where magic_variant is the low nibble of the magic (magic - 0x184D2A50, range 0..=15) and payload is a borrowed slice of the on-wire payload bytes (the skippable frame’s Frame_Size field worth of data) into input — no allocation.

The visitor sees skippable frames in stream order; interleaved regular zstd frames continue to decompress into output exactly as decode_all does.

output must be large enough to hold the decompressed data. Returns the number of bytes written to output.

Example

use structured_zstd::decoding::FrameDecoder;

let mut decoder = FrameDecoder::new();
let mut output = vec![0u8; 1024];
let mut collected: Vec<(u8, Vec<u8>)> = Vec::new();
let n = decoder.decode_all_with_skippable_visitor(
    input,
    &mut output,
    |variant, payload| collected.push((variant, payload.to_vec())),
)?;

pub fn decode_all_with_dict_handle( &mut self, input: &[u8], output: &mut [u8], dict: &DictionaryHandle, ) -> Result<usize, FrameDecoderError>

Decode multiple frames into the output slice using a pre-parsed dictionary handle.

input must contain an exact number of frames. Skippable frames are allowed and will be skipped during decode.

output must be large enough to hold the decompressed data. If you don’t know how large the output will be, use FrameDecoder::decode_blocks instead.

This calls FrameDecoder::init_with_dict_handle, and all bytes currently in the decoder will be lost.

Warning

Each decoded frame is initialized with dict, even when a frame header omits the optional dictionary ID. Callers must only use this API when they already know the input frames were encoded with the provided dictionary; otherwise decoded output can be silently corrupted.

pub fn decode_all_with_dict_bytes( &mut self, input: &[u8], output: &mut [u8], raw_dictionary: &[u8], ) -> Result<usize, FrameDecoderError>

Decode multiple frames into the output slice using a serialized dictionary.

Warning

Each decoded frame is initialized with the parsed dictionary, even when a frame header omits the optional dictionary ID. Callers must only use this API when they already know the input frames were encoded with that dictionary; otherwise decoded output can be silently corrupted.

pub fn decode_all_to_vec( &mut self, input: &[u8], output: &mut Vec<u8>, ) -> Result<(), FrameDecoderError>

Decode multiple frames into the extra capacity of the output vector.

input must contain an exact number of frames.

output must have enough spare capacity to hold the decompressed data. This adds no extra slack: exact-fit output is now eligible for the direct decode path, so a Vec::with_capacity(fcs) is decoded straight into without a growth/reallocation. It will NOT grow the vector to fit the decompressed payload itself; the caller’s pre-allocated capacity must already cover the data. If you don’t know how large the output will be, use FrameDecoder::decode_blocks instead.

This calls FrameDecoder::init, and all bytes currently in the decoder will be lost.

The length of the output vector is updated to include the decompressed data. The length is not changed if an error occurs.

Trait Implementations

impl Default for FrameDecoder

fn default() -> Self

Returns the “default value” for a type.

impl Read for FrameDecoder

Read bytes from the decode_buffer that are no longer needed. While the frame is not yet finished this will retain window_size bytes, else it will drain it completely

fn read(&mut self, target: &mut [u8]) -> Result<usize, Error>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Reads all bytes until EOF in this source, placing them into buf.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Reads all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Reads the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_, u8>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact( &mut self, cursor: BorrowedCursor<'_, u8>, ) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Reads the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Creates a “by reference” adapter for this instance of Read.

fn bytes(self) -> Bytes<Self>

where Self: Sized,

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>

where R: Read, Self: Sized,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

where Self: Sized,

Creates an adapter which will read at most limit bytes from it.

fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error>

where Self: Sized,

🔬This is a nightly-only experimental API. (read_array)

Read and return a fixed array of bytes from this source.