Struct StreamingDecoder 

pub struct StreamingDecoder<READ: Read, DEC: BorrowMut<FrameDecoder>> {
    pub decoder: DEC,
    /* private fields */
}

High level Zstandard frame decoder that can be used to decompress a given Zstandard frame.

This decoder implements io::Read, so you can interact with it by calling io::Read::read_to_end / io::Read::read_exact or passing this to another library / module as a source for the decoded content

If you need more control over how decompression takes place, you can use the lower level FrameDecoder, which allows for greater control over how decompression takes place but the implementor must call FrameDecoder::decode_blocks repeatedly to decode the entire frame.

Multiple frames

A stream is one or more frames (RFC 8878 3). Every read path, plain read and io::copy over it included, decodes them in turn, skips skippable frames wherever they stand, and applies the constructor’s dictionary to each frame. The source ending at a frame boundary ends the stream; bytes that do not form a frame are an error. read hands back a finished frame’s bytes before it reads the next frame’s header, so a source that stays open is not waited on while output is pending.

To decode a single frame and leave the bytes after it unread, drive a FrameDecoder directly.

When the source is read

Construction reads nothing; the first frame’s header is read by the first read or read_to_end, and a source that is not a zstd stream fails there, as upstream’s ZSTD_DStream reads nothing until it is fed. So into_inner hands back an untouched source when the decoder was never read, and the source is not lost when its first frame turns out to be damaged.

// `File` is std-only; `read_to_end` itself is available under no_std too.
#[cfg(feature = "std")]
{
    use std::fs::File;
    use std::io::Read;
    use structured_zstd::decoding::StreamingDecoder;

    // Read a Zstandard archive from the filesystem then decompress it into a vec.
    let mut f: File = todo!("Read a .zstd archive from somewhere");
    let mut decoder = StreamingDecoder::new(f);
    let mut result = Vec::new();
    Read::read_to_end(&mut decoder, &mut result).unwrap();
}

Fields

decoder: DEC

Implementations

impl<READ: Read, DEC: BorrowMut<FrameDecoder>> StreamingDecoder<READ, DEC>

pub fn new_with_decoder( source: READ, decoder: DEC, ) -> StreamingDecoder<READ, DEC>

new driving a caller-supplied FrameDecoder, so its buffers and settings (magicless format, registered dictionaries, checksum mode) carry over from stream to stream. Nothing is read from source until the first read.

Examples

use std::io::Read;
use structured_zstd::decoding::{FrameDecoder, StreamingDecoder};
use structured_zstd::encoding::{CompressionLevel, compress_to_vec};

let frame = compress_to_vec(&b"reused decoder"[..], CompressionLevel::Fastest);
let mut decoder = FrameDecoder::new();
for _ in 0..2 {
    let mut stream = StreamingDecoder::new_with_decoder(&frame[..], &mut decoder);
    let mut decoded = Vec::new();
    stream.read_to_end(&mut decoded).unwrap();
    assert_eq!(decoded, b"reused decoder");
}

pub fn new_with_decoder_and_dictionary_handle( source: READ, decoder: DEC, dict: &DictionaryHandle, ) -> Result<StreamingDecoder<READ, DEC>, DictionaryRefused<READ, DEC>>

new_with_decoder with dict applied to every frame even when its header omits the dictionary ID, as new_with_dictionary_handle applies it (same warning). A decoder reused frame after frame keeps its buffers and, for the same dictionary, the handle it already holds, so after the first frame it allocates nothing and touches no reference count.

Errors

Only for a dictionary whose content no frame can use, as a DictionaryRefused that hands back source, unread, and decoder as they were given. Nothing is read from source until the first read, which reports a frame that does not match the dictionary.

Examples

use std::io::Read;
use structured_zstd::decoding::{Dictionary, DictionaryHandle, FrameDecoder, StreamingDecoder};
use structured_zstd::encoding::{FrameCompressor, CompressionLevel};

let content = b"a dictionary of words the frames reuse".to_vec();
let dictionary = DictionaryHandle::from_dictionary(
    Dictionary::from_raw_content(1, content.clone()).unwrap(),
);
let mut compressor: FrameCompressor = FrameCompressor::new(CompressionLevel::Default);
compressor.set_dictionary(Dictionary::from_raw_content(1, content).unwrap()).unwrap();
let frame = compressor.compress_independent_frame(b"the frames reuse words");

let mut decoder = FrameDecoder::new();
for _ in 0..2 {
    let mut stream =
        StreamingDecoder::new_with_decoder_and_dictionary_handle(&frame[..], &mut decoder, &dictionary)
            .unwrap();
    let mut decoded = Vec::new();
    stream.read_to_end(&mut decoded).unwrap();
    assert_eq!(decoded, b"the frames reuse words");
}

// A dictionary no frame can use is refused, and the decoder and the
// frame come back to be used another way.
let mut unusable = Dictionary::from_raw_content(1, b"content".to_vec()).unwrap();
unusable.offset_hist[0] = 0;
let unusable = DictionaryHandle::from_dictionary(unusable);
let (source, decoder, _error) =
    StreamingDecoder::new_with_decoder_and_dictionary_handle(&frame[..], decoder, &unusable)
        .err()
        .expect("a zero repeat offset is refused")
        .into_parts();
assert_eq!(source, &frame[..]);

impl<READ: Read> StreamingDecoder<READ, FrameDecoder>

pub fn new(source: READ) -> StreamingDecoder<READ, FrameDecoder>

Create a streaming decoder over source.

Nothing is read from source here: the first frame’s header is read by the first read or read_to_end, which is where a source that is not a zstd stream, or is empty, reports its error. Until then into_inner hands source back untouched, and after such an error it hands back whatever the source holds past the bytes the decoder took.

Examples

use std::io::Read;
use structured_zstd::decoding::StreamingDecoder;

let mut decoder = StreamingDecoder::new(&b"not a zstd frame"[..]);
let mut out = Vec::new();
assert!(decoder.read_to_end(&mut out).is_err());

pub fn new_with_dictionary_handle( source: READ, dict: &DictionaryHandle, ) -> Result<StreamingDecoder<READ, FrameDecoder>, DictionaryRefused<READ>>

Create a streaming decoder using a pre-parsed dictionary handle.

Nothing is read from source here, as with new.

Errors

Only for a dictionary whose content no frame can use, as a DictionaryRefused that hands source back unread.

Warning

This constructor initializes the underlying FrameDecoder with dict, even if a frame header omits the optional dictionary ID. Callers must only use it when they already know the stream was encoded with this dictionary; otherwise decoded output can be silently corrupted.

pub fn new_with_dictionary_bytes( source: READ, raw_dictionary: &[u8], ) -> Result<StreamingDecoder<READ, FrameDecoder>, DictionaryRefused<READ>>

Create a streaming decoder using a serialized dictionary blob.

Errors

A blob that is not a dictionary, or one no frame can use, as a DictionaryRefused that hands source back unread.

Warning

This API forwards to StreamingDecoder::new_with_dictionary_handle and therefore applies the decoded dictionary to frames whose headers may omit the optional dictionary ID. Only use it when the stream is known to be encoded with that dictionary.

impl<READ: Read, DEC: BorrowMut<FrameDecoder>> StreamingDecoder<READ, DEC>

pub fn get_ref(&self) -> &READ

Gets a reference to the underlying reader.

pub fn get_mut(&mut self) -> &mut READ

Gets a mutable reference to the underlying reader.

It is inadvisable to directly read from the underlying reader.

pub fn decoder_mut(&mut self) -> &mut FrameDecoder

Gets a mutable reference to the frame decoder driving this stream.

Exposed for settings that are read as decoding proceeds rather than at construction — FrameDecoder::set_content_checksum above all, which a caller that wants mismatches to fail (rather than merely be computed) has to reach after the constructor has chosen the decoder, including on the dictionary paths.

pub fn into_inner(self) -> READ

where READ: Sized,

Destructures this object into the inner reader.

This reads nothing. A decoder that was never read hands the reader back untouched. The decoder reads only what its next step needs, so once a frame’s content has been delivered the reader stands right after that frame. Bytes of a step a WouldBlock left unfinished stay with the decoder.

Examples

use structured_zstd::decoding::StreamingDecoder;

let data = b"\x28\xb5\x2f\xfd and whatever follows";
let decoder = StreamingDecoder::new(&data[..]);
assert_eq!(decoder.into_inner(), &data[..]);

pub fn into_parts(self) -> (READ, DEC)

where READ: Sized,

Destructures this object into both the inner reader and FrameDecoder.

pub fn into_frame_decoder(self) -> DEC

Destructures this object into the inner FrameDecoder.

impl<READ: Read, DEC: BorrowMut<FrameDecoder>> StreamingDecoder<READ, DEC>

pub fn read_reporting_frames( &mut self, buf: &mut [u8], finished: impl FnMut(FinishedFrame), ) -> Result<usize, Error>

read, handing finished each frame the call finishes, at the moment it is verified and before the next frame is started, as upstream’s ZSTD_trace_decompress_end hook is called. One call can finish several frames (empty frames deliver nothing, so it passes on to the next), and each is reported in stream order; skippable frames are not. read_to_end reports none.

Examples

use structured_zstd::decoding::StreamingDecoder;
use structured_zstd::encoding::{CompressionLevel, compress_to_vec};

let frame = compress_to_vec(&b"one frame"[..], CompressionLevel::Fastest);
let mut decoder = StreamingDecoder::new(&frame[..]);
let mut buf = [0u8; 64];
let mut finished = Vec::new();
while decoder
    .read_reporting_frames(&mut buf, |frame| finished.push(frame))
    .unwrap()
    > 0
{}
assert_eq!(finished.len(), 1);
assert_eq!(finished[0].compressed_size, frame.len() as u64);
assert_eq!(finished[0].decompressed_size, 9);

Trait Implementations

impl<READ: Read, DEC: BorrowMut<FrameDecoder>> Read for StreamingDecoder<READ, DEC>

fn read(&mut self, buf: &mut [u8]) -> Result<usize, Error>

Decode the stream into buf, frame after frame (RFC 8878 3: a stream is one or more frames), as upstream zstd’s ZSTD_decompressStream does. Skippable frames are skipped; the source ending at a frame boundary is the end of the stream, anything else is an error.

fn read_to_end(&mut self, output: &mut Vec<u8>) -> Result<usize, Error>

Decode-in-place fast path for whole-frame consumption. Instead of the generic read loop (decode block -> RingBuffer -> copy into the caller buffer), buffer the (compressed, hence small) source and decode STRAIGHT into output’s spare capacity via the single-copy direct path, pre-sized from the frame’s declared content size. Only taken when the decoder is at a frame boundary (nothing partially decoded / undrained); otherwise it falls back to the generic grow-and-read loop so a caller that mixed read with read_to_end still gets correct output.

Per the Read::read_to_end contract this consumes the source to EOF: if the stream holds several concatenated frames they are ALL decoded (and skippable frames skipped), as with read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Reads all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Reads the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_, u8>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact( &mut self, cursor: BorrowedCursor<'_, u8>, ) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Reads the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Creates a “by reference” adapter for this instance of Read.

fn bytes(self) -> Bytes<Self>

where Self: Sized,

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>

where R: Read, Self: Sized,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

where Self: Sized,

Creates an adapter which will read at most limit bytes from it.

fn read_array<const N: usize>(&mut self) -> Result<[u8; N], Error>

where Self: Sized,

🔬This is a nightly-only experimental API. (read_array)

Read and return a fixed array of bytes from this source.

fn read_le<T>(&mut self) -> Result<T, Error>

where T: FromEndianBytes, Self: Sized,

🔬This is a nightly-only experimental API. (read_le)

Read and return a type (e.g. an integer) in little-endian order.

fn read_be<T>(&mut self) -> Result<T, Error>

where T: FromEndianBytes, Self: Sized,

🔬This is a nightly-only experimental API. (read_le)

Read and return a type (e.g. an integer) in big-endian order.