Struct ruzstd::frame_decoder::FrameDecoder

pub struct FrameDecoder { /* fields omitted */ }

This implements a decoder for zstd frames. This decoder is able to decode frames only partially and gives control over how many bytes/blocks will be decoded at a time (so you dont have to decode a 10GB file into memory all at once). It reads bytes as needed from a provided source and can be read from to collect partial results.

If you want to just read the whole frame with an io::Read without having to deal with manually calling decode_blocks you can use the provided StreamingDecoder with wraps this FrameDecoder

Workflow is as follows:

use ruzstd::frame_decoder::BlockDecodingStrategy;
use std::io::Read;
use std::io::Write;


fn decode_this(file: &mut std::io::Read) {
    //Create a new decoder
    let mut frame_dec = ruzstd::FrameDecoder::new();
    let mut result = Vec::new();
 
    // Use reset or init to make the decoder ready to decocde the frame from the io::Read
    frame_dec.reset(file).unwrap();
 
    // Loop until the frame has been decoded completely
    while !frame_dec.is_finished() {
        // decode (roughly) batch_size many bytes
        frame_dec.decode_blocks(file, BlockDecodingStrategy::UptoBytes(1024)).unwrap();
 
        // read from the decoder to collect bytes from the internal buffer
        let bytes_read = frame_dec.read(result.as_mut_slice()).unwrap();
         
        // then do something with it
        do_something(&result[0..bytes_read]);
    }
 
    // handle the last chunk of data
    while frame_dec.can_collect() > 0 {
        let x = frame_dec.read(result.as_mut_slice()).unwrap();
 
        do_something(&result[0..x]);
    }
}
 
fn do_something(data: &[u8]) {
    std::io::stdout().write_all(data).unwrap();
}

Methods

impl FrameDecoder

Important traits for FrameDecoder

impl Read for FrameDecoder

pub fn new() -> FrameDecoder

This will create a new decoder without allocating anything yet. init()/reset() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations

pub fn init(&mut self, source: &mut dyn Read) -> Result<(), String>

init() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations

Note that all bytes currently in the decodebuffer from any previous frame will be lost. Collect them with collect()/collect_to_writer()

equivalent to reset()

pub fn init_with_dict(
    &mut self,
    source: &mut dyn Read,
    dict: &[u8]
) -> Result<(), String>

Like init but provides the dict to use for the next frame

pub fn reset(&mut self, source: &mut dyn Read) -> Result<(), String>

reset() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations

Note that all bytes currently in the decodebuffer from any previous frame will be lost. Collect them with collect()/collect_to_writer()

equivalent to init()

pub fn reset_with_dict(
    &mut self,
    source: &mut dyn Read,
    dict: &[u8]
) -> Result<(), String>

Like reset but provides the dict to use for the next frame

pub fn add_dict(&mut self, raw_dict: &[u8]) -> Result<(), String>

Add a dict to the FrameDecoder that can be used when needed. The FrameDecoder uses the appropriate one dynamically

pub fn content_size(&self) -> Option<u64>

Returns how many bytes the frame contains after decompression

pub fn get_checksum_from_data(&self) -> Option<u32>

Returns the checksum that was read from the data. Only available after all bytes have been read. It is the last 4 bytes of a zstd-frame

pub fn get_calculated_checksum(&self) -> Option<u32>

Returns the checksum that was calculated while decoding. Only a sensible value after all decoded bytes have been collected/read from the FrameDecoder

pub fn bytes_read_from_source(&self) -> u64

Counter for how many bytes have been consumed while deocidng the frame

pub fn is_finished(&self) -> bool

Whether the current frames last block has been decoded yet If this returns true you can call the drain* functions to get all content (the read() function will drain automatically if this returns true)

pub fn blocks_decoded(&self) -> usize

Counter for how many blocks have already been decoded

pub fn decode_blocks(
    &mut self,
    source: &mut dyn Read,
    strat: BlockDecodingStrategy
) -> Result<bool, FrameDecoderError>

Decodes blocks from a reader. It requires that the framedecoder has been initialized first. The Strategy influences how many blocks will be decoded before the function returns This is important if you want to manage memory consumption carefully. If you dont care about that you can just choose the strategy "All" and have all blocks of the frame decoded into the buffer

pub fn collect(&mut self) -> Option<Vec<u8>>

Collect bytes and retain window_size bytes while decoding is still going on. After decoding of the frame (is_finished() == true) has finished it will collect all remaining bytes

pub fn collect_to_writer(&mut self, w: &mut dyn Write) -> Result<usize, Error>

Collect bytes and retain window_size bytes while decoding is still going on. After decoding of the frame (is_finished() == true) has finished it will collect all remaining bytes

pub fn can_collect(&self) -> usize

How many bytes can currently be collected from the decodebuffer, while decoding is going on this will be lower than the ectual decodbuffer size because window_size bytes need to be retained for decoding. After decoding of the frame (is_finished() == true) has finished it will report all remaining bytes

pub fn decode_from_to(
    &mut self,
    source: &[u8],
    target: &mut [u8]
) -> Result<(usize, usize), FrameDecoderError>

Decodes as many blocks as possible from the source slice and reads from the decodebuffer into the target slice The source slice may contain only parts of a frame but must contain at least one full block to make progress

By all means use decode_blocks if you have a io.Reader available. This is just for compatibility with other decompressors which try to serve an old-style c api

Returns (read, written), if read == 0 then the source did not contain a full block and further calls with the same input will not make any progress!

Note that no kind of block can be bigger than 128kb. So to be safe use at least 128*1024 (max block content size) + 3 (block_header size) + 18 (max frame_header size) bytes as your source buffer

You may call this function with an empty source after all bytes have been decoded. This is equivalent to just call decoder.read(&mut traget)

Trait Implementations

impl Default for FrameDecoder

fn default() -> Self

Returns the "default value" for a type.

impl Read for FrameDecoder

Read bytes from the decode_buffer that are no longer needed. While the frame is not yet finished this will retain window_size bytes, else it will drain it completely

fn read(&mut self, target: &mut [u8]) -> Result<usize, Error>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

unsafe fn initializer(&self) -> Initializer

🔬 This is a nightly-only experimental API. (read_initializer)

Determines if this Reader can work with buffers of uninitialized memory.

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Read all bytes until EOF in this source, placing them into buf.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Read all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Read the exact number of bytes required to fill buf.

fn by_ref(&mut self) -> &mut Self

Creates a "by reference" adaptor for this instance of Read.

fn bytes(self) -> Bytes<Self>

Transforms this Read instance to an [Iterator] over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R> where
    R: Read, 

Creates an adaptor which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

Creates an adaptor which will read at most limit bytes from it.