Skip to main content

StreamingEncoder

Struct StreamingEncoder 

Source
pub struct StreamingEncoder<'a> { /* private fields */ }
Expand description

Streaming encoder that consumes input in chunks and produces compressed output progressively.

Call push to feed input and take_output to drain produced bytes; finalize the frame with finish, then optionally reset to begin a new frame on the same context (preserving allocations and dictionary).

§Footprint

Roughly 25 KB by value. The bulk is the Huffman compression workspace and the literals encoding state, held as inline arrays so the per-block path reaches them without an indirection and reuses them across blocks and frames. The cost is that the type is unpleasant to embed: it will trip clippy::large_enum_variant in an enum, and it enlarges any future holding it across an await. Box it when it is not a local. Everything wrapping an encoder inherits this, Writer included; StreamingDecoder is a few hundred bytes and needs none of it.

Implementations§

Source§

impl StreamingEncoder<'static>

Source

pub fn new(options: EncoderOptions) -> Result<Self>

Construct a dictionary-less streaming encoder with the given options. Writes the frame header into the output buffer immediately.

Source§

impl<'a> StreamingEncoder<'a>

Source

pub const RECOMMENDED_INPUT_SIZE: usize = BLOCK_SIZE_MAX

Input chunk size that keeps the encoder working on whole blocks, upstream’s ZSTD_CStreamInSize.

Nothing requires it: push buffers whatever it is given and encodes a block once one is complete. Pushing this much at a time simply means no byte waits in that buffer for a later call – except on a stream whose pledge is its whole block, which holds that block back for finish to flag; see push.

Source

pub const RECOMMENDED_OUTPUT_SIZE: usize

Output buffer size that can always take one complete block in a single read, upstream’s ZSTD_CStreamOutSize.

The worst case it covers is a block that would not compress: the encoder emits it raw, so the payload is the full block, and the frame header and the trailing checksum both have to fit alongside it. A 128 KiB block may also be split into 8 KiB sub-blocks, each paying its own 3-byte header, which is the middle term. This is compress_bound of one block, as a constant.

A smaller buffer is not an error; read just returns short and the remainder stays queued.

Source

pub fn with_dict(dict: &'a [u8], options: EncoderOptions) -> Result<Self>

Construct a streaming encoder that uses dict as a dictionary. The dictionary bytes are parsed once during construction.

Source

pub fn with_prepared_dict( dict: &EncoderDictionary<'a>, options: EncoderOptions, ) -> Result<Self>

Construct a streaming encoder that shares an already-parsed dictionary. The clone is cheap and reuses any cached parser-built tables.

Source

pub fn push(&mut self, src: &[u8]) -> Result<()>

Append src to the input stream. Complete blocks are encoded immediately; the trailing partial block stays buffered until enough input arrives or flush / finish runs. Returns an error if called after finish without an intervening reset.

One stream produces nothing here at all. The block size is capped at pledged_src_size, so a pledge that fits in one block – at default parameters, any pledge of 128 KiB or less – is the block size, and the whole content then waits for finish, so that the single block carries the last-block flag itself instead of trailing an empty block behind it. A caller pumping read between pushes sees the frame header and then nothing until it finishes. This is upstream’s behavior under the same pledge, and it makes the frame identical to encode_all_with_options’.

Source

pub fn flush(&mut self) -> Result<()>

Encode any buffered partial block as a non-final block, making its output bytes available via take_output. Useful for forcing progress on slow producers.

Source

pub fn finish(&mut self) -> Result<()>

Emit the final block (marked as last) and, if checksums are enabled, the four-byte content checksum. After this returns the frame is complete and push/flush will fail until reset runs.

Source

pub fn reset(&mut self) -> Result<()>

Begin a fresh frame on the same context, preserving allocations and the configured dictionary. Must be called only after finish.

Source

pub fn take_output(&mut self) -> Vec<u8> ⓘ

Hand back the bytes produced so far and reset the internal output buffer to empty. Safe to call at any time; combine with push / finish to drain output incrementally.

This hands over the buffer itself, so the encoder starts the next block with no output capacity and grows a fresh allocation for it. That is the right trade when the caller wants to own the Vec, and the wrong one for a pump that drains every block into somewhere else: read and pending_output reuse the buffer instead and allocate nothing per block.

Source

pub fn read(&mut self, dst: &mut [u8]) -> usize

Copy up to dst.len() compressed bytes into dst and return how many were copied. Bytes that are read are removed from the internal buffer, which keeps its capacity for the next block.

This is the counterpart to StreamingDecoder::read, and the closest thing here to handing upstream a ZSTD_outBuffer: a pump built on it allocates once for dst and never again.

use zstandard::{EncoderOptions, StreamingEncoder, decode_all};

let payload = b"streamed into a fixed buffer".repeat(4_000);
let mut encoder = StreamingEncoder::new(EncoderOptions::default())?;
let mut window = vec![0u8; StreamingEncoder::RECOMMENDED_OUTPUT_SIZE];
let mut compressed = Vec::new();

for chunk in payload.chunks(StreamingEncoder::RECOMMENDED_INPUT_SIZE) {
    encoder.push(chunk)?;
    loop {
        let n = encoder.read(&mut window);
        if n == 0 { break; }
        compressed.extend_from_slice(&window[..n]);
    }
}
encoder.finish()?;
loop {
    let n = encoder.read(&mut window);
    if n == 0 { break; }
    compressed.extend_from_slice(&window[..n]);
}

assert_eq!(decode_all(&compressed)?, payload);
Source

pub fn pending_output(&self) -> &[u8] ⓘ

Borrow the compressed bytes buffered so far without removing them.

Pair with consume_output to forward output somewhere that takes a slice, with no copy in between and no allocation per block. This is what io::Writer uses.

Source

pub fn consume_output(&mut self, count: usize)

Discard the first count bytes of pending_output.

§Panics

If count exceeds pending_output_len. Consuming more than was produced is a caller bug, and silently clamping it would drop compressed bytes and leave a truncated frame that still parses as a frame.

Source

pub fn pending_output_len(&self) -> usize

Number of compressed bytes currently waiting to be drained by take_output.

Source

pub fn is_finished(&self) -> bool

true if finish has been called and no reset has followed it.

Trait Implementations§

Source§

impl Debug for StreamingEncoder<'_>

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for StreamingEncoder<'static>

Source§

fn default() -> Self

Returns the “default value” for a type. Read more

Auto Trait Implementations§

§

impl<'a> Freeze for StreamingEncoder<'a>

§

impl<'a> RefUnwindSafe for StreamingEncoder<'a>

§

impl<'a> Send for StreamingEncoder<'a>

§

impl<'a> Sync for StreamingEncoder<'a>

§

impl<'a> Unpin for StreamingEncoder<'a>

§

impl<'a> UnsafeUnpin for StreamingEncoder<'a>

§

impl<'a> UnwindSafe for StreamingEncoder<'a>

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.