Struct ruzstd::frame_decoder::FrameDecoder [−][src]
pub struct FrameDecoder { /* fields omitted */ }
Expand description
This implements a decoder for zstd frames. This decoder is able to decode frames only partially and gives control over how many bytes/blocks will be decoded at a time (so you dont have to decode a 10GB file into memory all at once). It reads bytes as needed from a provided source and can be read from to collect partial results.
If you want to just read the whole frame with an io::Read without having to deal with manually calling decode_blocks you can use the provided StreamingDecoder with wraps this FrameDecoder
Workflow is as follows:
use ruzstd::frame_decoder::BlockDecodingStrategy; use std::io::Read; use std::io::Write; fn decode_this(file: &mut std::io::Read) { //Create a new decoder let mut frame_dec = ruzstd::FrameDecoder::new(); let mut result = Vec::new(); // Use reset or init to make the decoder ready to decocde the frame from the io::Read frame_dec.reset(file).unwrap(); // Loop until the frame has been decoded completely while !frame_dec.is_finished() { // decode (roughly) batch_size many bytes frame_dec.decode_blocks(file, BlockDecodingStrategy::UptoBytes(1024)).unwrap(); // read from the decoder to collect bytes from the internal buffer let bytes_read = frame_dec.read(result.as_mut_slice()).unwrap(); // then do something with it do_something(&result[0..bytes_read]); } // handle the last chunk of data while frame_dec.can_collect() > 0 { let x = frame_dec.read(result.as_mut_slice()).unwrap(); do_something(&result[0..x]); } } fn do_something(data: &[u8]) { std::io::stdout().write_all(data).unwrap(); }
Implementations
This will create a new decoder without allocating anything yet. init()/reset() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations
init() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations
Note that all bytes currently in the decodebuffer from any previous frame will be lost. Collect them with collect()/collect_to_writer()
equivalent to reset()
Like init but provides the dict to use for the next frame
reset() will allocate all needed buffers if it is the first time this decoder is used else they just reset these buffers with not further allocations
Note that all bytes currently in the decodebuffer from any previous frame will be lost. Collect them with collect()/collect_to_writer()
equivalent to init()
Like reset but provides the dict to use for the next frame
Add a dict to the FrameDecoder that can be used when needed. The FrameDecoder uses the appropriate one dynamically
Returns how many bytes the frame contains after decompression
Returns the checksum that was read from the data. Only available after all bytes have been read. It is the last 4 bytes of a zstd-frame
Returns the checksum that was calculated while decoding. Only a sensible value after all decoded bytes have been collected/read from the FrameDecoder
Counter for how many bytes have been consumed while deocidng the frame
Whether the current frames last block has been decoded yet If this returns true you can call the drain* functions to get all content (the read() function will drain automatically if this returns true)
Counter for how many blocks have already been decoded
pub fn decode_blocks(
&mut self,
source: &mut dyn Read,
strat: BlockDecodingStrategy
) -> Result<bool, FrameDecoderError>
pub fn decode_blocks(
&mut self,
source: &mut dyn Read,
strat: BlockDecodingStrategy
) -> Result<bool, FrameDecoderError>
Decodes blocks from a reader. It requires that the framedecoder has been initialized first. The Strategy influences how many blocks will be decoded before the function returns This is important if you want to manage memory consumption carefully. If you dont care about that you can just choose the strategy “All” and have all blocks of the frame decoded into the buffer
Collect bytes and retain window_size bytes while decoding is still going on. After decoding of the frame (is_finished() == true) has finished it will collect all remaining bytes
Collect bytes and retain window_size bytes while decoding is still going on. After decoding of the frame (is_finished() == true) has finished it will collect all remaining bytes
How many bytes can currently be collected from the decodebuffer, while decoding is going on this will be lower than the ectual decodbuffer size because window_size bytes need to be retained for decoding. After decoding of the frame (is_finished() == true) has finished it will report all remaining bytes
Decodes as many blocks as possible from the source slice and reads from the decodebuffer into the target slice The source slice may contain only parts of a frame but must contain at least one full block to make progress
By all means use decode_blocks if you have a io.Reader available. This is just for compatibility with other decompressors which try to serve an old-style c api
Returns (read, written), if read == 0 then the source did not contain a full block and further calls with the same input will not make any progress!
Note that no kind of block can be bigger than 128kb. So to be safe use at least 128*1024 (max block content size) + 3 (block_header size) + 18 (max frame_header size) bytes as your source buffer
You may call this function with an empty source after all bytes have been decoded. This is equivalent to just call decoder.read(&mut traget)
Trait Implementations
Read bytes from the decode_buffer that are no longer needed. While the frame is not yet finished this will retain window_size bytes, else it will drain it completely
Pull some bytes from this source into the specified buffer, returning how many bytes were read. Read more
Like read
, except that it reads into a slice of buffers. Read more
can_vector
)Determines if this Read
er has an efficient read_vectored
implementation. Read more
read_initializer
)Determines if this Read
er can work with buffers of uninitialized
memory. Read more
Read all bytes until EOF in this source, placing them into buf
. Read more
Read all bytes until EOF in this source, appending them to buf
. Read more
Read the exact number of bytes required to fill buf
. Read more
Creates a “by reference” adaptor for this instance of Read
. Read more
Creates an adaptor which will chain this stream with another. Read more
Auto Trait Implementations
impl RefUnwindSafe for FrameDecoder
impl Send for FrameDecoder
impl Sync for FrameDecoder
impl Unpin for FrameDecoder
impl UnwindSafe for FrameDecoder
Blanket Implementations
Mutably borrows from an owned value. Read more
Reads an unsigned 8 bit integer from the underlying reader. Read more
Reads a signed 8 bit integer from the underlying reader. Read more
Reads an unsigned 16 bit integer from the underlying reader. Read more
Reads a signed 16 bit integer from the underlying reader. Read more
Reads an unsigned 24 bit integer from the underlying reader. Read more
Reads a signed 24 bit integer from the underlying reader. Read more
Reads an unsigned 32 bit integer from the underlying reader. Read more
Reads a signed 32 bit integer from the underlying reader. Read more
Reads an unsigned 48 bit integer from the underlying reader. Read more
Reads a signed 48 bit integer from the underlying reader. Read more
Reads an unsigned 64 bit integer from the underlying reader. Read more
Reads a signed 64 bit integer from the underlying reader. Read more
Reads an unsigned 128 bit integer from the underlying reader. Read more
Reads a signed 128 bit integer from the underlying reader. Read more
Reads an unsigned n-bytes integer from the underlying reader. Read more
Reads a signed n-bytes integer from the underlying reader. Read more
Reads an unsigned n-bytes integer from the underlying reader.
Reads a signed n-bytes integer from the underlying reader.
Reads a IEEE754 single-precision (4 bytes) floating point number from the underlying reader. Read more
Reads a IEEE754 double-precision (8 bytes) floating point number from the underlying reader. Read more
Reads a sequence of unsigned 16 bit integers from the underlying reader. Read more
Reads a sequence of unsigned 32 bit integers from the underlying reader. Read more
Reads a sequence of unsigned 64 bit integers from the underlying reader. Read more
Reads a sequence of unsigned 128 bit integers from the underlying reader. Read more
Reads a sequence of signed 8 bit integers from the underlying reader. Read more
Reads a sequence of signed 16 bit integers from the underlying reader. Read more
Reads a sequence of signed 32 bit integers from the underlying reader. Read more
Reads a sequence of signed 64 bit integers from the underlying reader. Read more
Reads a sequence of signed 128 bit integers from the underlying reader. Read more
Reads a sequence of IEEE754 single-precision (4 bytes) floating point numbers from the underlying reader. Read more
Reads a sequence of IEEE754 double-precision (8 bytes) floating point numbers from the underlying reader. Read more