Struct FrameDecoder 

pub struct FrameDecoder { /* private fields */ }

Low level Zstandard decoder that can be used to decompress frames with fine control over when and how many bytes are decoded.

This decoder is able to decode frames only partially and gives control over how many bytes/blocks will be decoded at a time (so you don’t have to decode a 10GB file into memory all at once). It reads bytes as needed from a provided source and can be read from to collect partial results.

If you want to just read the whole frame with an io::Read without having to deal with manually calling FrameDecoder::decode_blocks you can use the provided crate::decoding::StreamingDecoder wich wraps this FrameDecoder.

Workflow is as follows:

use structured_zstd::decoding::BlockDecodingStrategy;

use std::io::{Read, Write};

// no_std environments can use the crate's own Read traits
use structured_zstd::io::{Read, Write};

fn decode_this(mut file: impl Read) {
    //Create a new decoder
    let mut frame_dec = structured_zstd::decoding::FrameDecoder::new();
    let mut result = Vec::new();

    // Use reset or init to make the decoder ready to decode the frame from the io::Read
    frame_dec.reset(&mut file).unwrap();

    // Loop until the frame has been decoded completely
    while !frame_dec.is_finished() {
        // decode (roughly) batch_size many bytes
        frame_dec.decode_blocks(&mut file, BlockDecodingStrategy::UptoBytes(1024)).unwrap();

        // read from the decoder to collect bytes from the internal buffer
        let bytes_read = frame_dec.read(result.as_mut_slice()).unwrap();

        // then do something with it
        do_something(&result[0..bytes_read]);
    }

    // handle the last chunk of data
    while frame_dec.can_collect() > 0 {
        let x = frame_dec.read(result.as_mut_slice()).unwrap();

        do_something(&result[0..x]);
    }
}

fn do_something(data: &[u8]) {
    std::io::stdout().write_all(data).unwrap();
}

Implementations

impl FrameDecoder

pub fn new() -> FrameDecoder

This will create a new decoder without allocating anything yet. init()/reset() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations

pub fn init(&mut self, source: impl Read) -> Result<(), FrameDecoderError>

init() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations

Note that all bytes currently in the decodebuffer from any previous frame will be lost. Collect them with collect()/collect_to_writer()

equivalent to reset()

pub fn init_with_dict_handle( &mut self, source: impl Read, dict: &DictionaryHandle, ) -> Result<(), FrameDecoderError>

Initialize the decoder for a new frame using a pre-parsed dictionary handle.

If the frame header has a dictionary ID, this validates it against dict.id() and returns FrameDecoderError::DictIdMismatch on mismatch.

If the header omits the optional dictionary ID, this still applies the provided dictionary handle.

Warning

This method always applies dict unless the frame header contains a non-matching dictionary ID. Callers must only use this API when they already know the frame was encoded with the provided dictionary, even if the frame header omits the dictionary ID or encodes an explicit dictionary ID of 0.

Passing a dictionary for a frame that was not encoded with it can silently corrupt the decoded output.

pub fn reset(&mut self, source: impl Read) -> Result<(), FrameDecoderError>

reset() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations

Note that all bytes currently in the decodebuffer from any previous frame will be lost. Collect them with collect()/collect_to_writer()

equivalent to init()

pub fn reset_with_dict_handle( &mut self, source: impl Read, dict: &DictionaryHandle, ) -> Result<(), FrameDecoderError>

Reset this decoder for a new frame using a pre-parsed dictionary handle.

If the frame header has a dictionary ID, this validates it against dict.id() and returns FrameDecoderError::DictIdMismatch on mismatch.

If the header omits the optional dictionary ID, this still applies the provided dictionary handle.

Warning

This method always applies dict unless the frame header contains a non-matching dictionary ID. Callers must only use this API when they already know the frame was encoded with the provided dictionary, even if the frame header omits the dictionary ID or encodes an explicit dictionary ID of 0.

Passing a dictionary for a frame that was not encoded with it can silently corrupt the decoded output.

pub fn add_dict(&mut self, dict: Dictionary) -> Result<(), FrameDecoderError>

Add a dictionary that can be selected dynamically by frame dictionary ID.

Returns FrameDecoderError::DictAlreadyRegistered if the ID is already registered (either as owned or shared).

pub fn add_dict_from_bytes( &mut self, raw_dictionary: &[u8], ) -> Result<(), FrameDecoderError>

Parse and add a serialized dictionary blob.

pub fn add_dict_handle( &mut self, dict: DictionaryHandle, ) -> Result<(), FrameDecoderError>

Add a pre-parsed dictionary handle for reuse across decoders.

This API is available on targets with pointer-width atomics (target_has_atomic = "ptr").

Returns FrameDecoderError::DictAlreadyRegistered if the ID is already registered (either as owned or shared).

pub fn force_dict(&mut self, dict_id: u32) -> Result<(), FrameDecoderError>

pub fn content_size(&self) -> u64

Returns how many bytes the frame contains after decompression

pub fn get_checksum_from_data(&self) -> Option<u32>

Returns the checksum that was read from the data. Only available after all bytes have been read. It is the last 4 bytes of a zstd-frame

pub fn get_calculated_checksum(&self) -> Option<u32>

Returns the checksum that was calculated while decoding. Only a sensible value after all decoded bytes have been collected/read from the FrameDecoder

pub fn bytes_read_from_source(&self) -> u64

Counter for how many bytes have been consumed while decoding the frame

pub fn is_finished(&self) -> bool

Whether the current frames last block has been decoded yet If this returns true you can call the drain* functions to get all content (the read() function will drain automatically if this returns true)

pub fn blocks_decoded(&self) -> usize

Counter for how many blocks have already been decoded

pub fn decode_blocks( &mut self, source: impl Read, strat: BlockDecodingStrategy, ) -> Result<bool, FrameDecoderError>

Decodes blocks from a reader. It requires that the framedecoder has been initialized first. The Strategy influences how many blocks will be decoded before the function returns This is important if you want to manage memory consumption carefully. If you don’t care about that you can just choose the strategy “All” and have all blocks of the frame decoded into the buffer

pub fn collect(&mut self) -> Option<Vec<u8>>

Collect bytes and retain window_size bytes while decoding is still going on. After decoding of the frame (is_finished() == true) has finished it will collect all remaining bytes

pub fn collect_to_writer(&mut self, w: impl Write) -> Result<usize, Error>

Collect bytes and retain window_size bytes while decoding is still going on. After decoding of the frame (is_finished() == true) has finished it will collect all remaining bytes

pub fn can_collect(&self) -> usize

How many bytes can currently be collected from the decodebuffer, while decoding is going on this will be lower than the actual decodbuffer size because window_size bytes need to be retained for decoding. After decoding of the frame (is_finished() == true) has finished it will report all remaining bytes

pub fn decode_from_to( &mut self, source: &[u8], target: &mut [u8], ) -> Result<(usize, usize), FrameDecoderError>

Decodes as many blocks as possible from the source slice and reads from the decodebuffer into the target slice The source slice may contain only parts of a frame but must contain at least one full block to make progress

By all means use decode_blocks if you have a io.Reader available. This is just for compatibility with other decompressors which try to serve an old-style c api

Returns (read, written), if read == 0 then the source did not contain a full block and further calls with the same input will not make any progress!

Note that no kind of block can be bigger than 128kb. So to be safe use at least 128*1024 (max block content size) + 3 (block_header size) + 18 (max frame_header size) bytes as your source buffer

You may call this function with an empty source after all bytes have been decoded. This is equivalent to just call decoder.read(&mut target)

pub fn decode_all( &mut self, input: &[u8], output: &mut [u8], ) -> Result<usize, FrameDecoderError>

Decode multiple frames into the output slice.

input must contain an exact number of frames. Skippable frames are allowed and will be skipped during decode.

output must be large enough to hold the decompressed data. If you don’t know how large the output will be, use FrameDecoder::decode_blocks instead.

This calls FrameDecoder::init, and all bytes currently in the decoder will be lost.

Returns the number of bytes written to output.

pub fn decode_all_with_dict_handle( &mut self, input: &[u8], output: &mut [u8], dict: &DictionaryHandle, ) -> Result<usize, FrameDecoderError>

Decode multiple frames into the output slice using a pre-parsed dictionary handle.

input must contain an exact number of frames. Skippable frames are allowed and will be skipped during decode.

output must be large enough to hold the decompressed data. If you don’t know how large the output will be, use FrameDecoder::decode_blocks instead.

This calls FrameDecoder::init_with_dict_handle, and all bytes currently in the decoder will be lost.

Warning

Each decoded frame is initialized with dict, even when a frame header omits the optional dictionary ID. Callers must only use this API when they already know the input frames were encoded with the provided dictionary; otherwise decoded output can be silently corrupted.

pub fn decode_all_with_dict_bytes( &mut self, input: &[u8], output: &mut [u8], raw_dictionary: &[u8], ) -> Result<usize, FrameDecoderError>

Decode multiple frames into the output slice using a serialized dictionary.

Warning

Each decoded frame is initialized with the parsed dictionary, even when a frame header omits the optional dictionary ID. Callers must only use this API when they already know the input frames were encoded with that dictionary; otherwise decoded output can be silently corrupted.

pub fn decode_all_to_vec( &mut self, input: &[u8], output: &mut Vec<u8>, ) -> Result<(), FrameDecoderError>

Decode multiple frames into the extra capacity of the output vector.

input must contain an exact number of frames.

output must have enough extra capacity to hold the decompressed data. This function will not reallocate or grow the vector. If you don’t know how large the output will be, use FrameDecoder::decode_blocks instead.

This calls FrameDecoder::init, and all bytes currently in the decoder will be lost.

The length of the output vector is updated to include the decompressed data. The length is not changed if an error occurs.

Trait Implementations

impl Default for FrameDecoder

fn default() -> Self

Returns the “default value” for a type.

impl Read for FrameDecoder

Read bytes from the decode_buffer that are no longer needed. While the frame is not yet finished this will retain window_size bytes, else it will drain it completely

fn read(&mut self, target: &mut [u8]) -> Result<usize, Error>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Reads all bytes until EOF in this source, placing them into buf.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Reads all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Reads the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact(&mut self, cursor: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Reads the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Creates a “by reference” adapter for this instance of Read.

fn bytes(self) -> Bytes<Self>

where Self: Sized,

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>

where R: Read, Self: Sized,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

where Self: Sized,

Creates an adapter which will read at most limit bytes from it.

fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error>

where Self: Sized,

🔬This is a nightly-only experimental API. (read_array)

Read and return a fixed array of bytes from this source.