pub struct StreamingEncoder<'a> { /* private fields */ }Expand description
Streaming encoder that consumes input in chunks and produces compressed output progressively.
Call push to feed input and take_output
to drain produced bytes; finalize the frame with finish,
then optionally reset to begin a new frame on the same
context (preserving allocations and dictionary).
§Footprint
Roughly 25 KB by value. The bulk is the Huffman compression workspace and the
literals encoding state, held as inline arrays so the per-block path reaches
them without an indirection and reuses them across blocks and frames. The
cost is that the type is unpleasant to embed: it will trip
clippy::large_enum_variant in an enum, and it enlarges any future holding
it across an await. Box it when it is not a local. Everything wrapping an
encoder inherits this, Writer included;
StreamingDecoder is a few hundred bytes and needs none of it.
Implementations§
Source§impl StreamingEncoder<'static>
impl StreamingEncoder<'static>
Sourcepub fn new(options: EncoderOptions) -> Result<Self>
pub fn new(options: EncoderOptions) -> Result<Self>
Construct a dictionary-less streaming encoder with the given options. Writes the frame header into the output buffer immediately.
Source§impl<'a> StreamingEncoder<'a>
impl<'a> StreamingEncoder<'a>
Sourcepub const RECOMMENDED_INPUT_SIZE: usize = BLOCK_SIZE_MAX
pub const RECOMMENDED_INPUT_SIZE: usize = BLOCK_SIZE_MAX
Input chunk size that keeps the encoder working on whole blocks,
upstream’s ZSTD_CStreamInSize.
Nothing requires it: push buffers whatever it is given
and encodes a block once one is complete. Pushing this much at a time
simply means no byte waits in that buffer for a later call – except on
a stream whose pledge is its whole block, which holds that block back
for finish to flag; see push.
Sourcepub const RECOMMENDED_OUTPUT_SIZE: usize
pub const RECOMMENDED_OUTPUT_SIZE: usize
Output buffer size that can always take one complete block in a single
read, upstream’s ZSTD_CStreamOutSize.
The worst case it covers is a block that would not compress: the
encoder emits it raw, so the payload is the full block, and the frame
header and the trailing checksum both have to fit alongside it. A
128 KiB block may also be split into 8 KiB sub-blocks, each paying its
own 3-byte header, which is the middle term. This is
compress_bound of one block, as a constant.
A smaller buffer is not an error; read just returns short and the
remainder stays queued.
Sourcepub fn with_dict(dict: &'a [u8], options: EncoderOptions) -> Result<Self>
pub fn with_dict(dict: &'a [u8], options: EncoderOptions) -> Result<Self>
Construct a streaming encoder that uses dict as a dictionary. The
dictionary bytes are parsed once during construction.
Sourcepub fn with_prepared_dict(
dict: &EncoderDictionary<'a>,
options: EncoderOptions,
) -> Result<Self>
pub fn with_prepared_dict( dict: &EncoderDictionary<'a>, options: EncoderOptions, ) -> Result<Self>
Construct a streaming encoder that shares an already-parsed dictionary. The clone is cheap and reuses any cached parser-built tables.
Sourcepub fn push(&mut self, src: &[u8]) -> Result<()>
pub fn push(&mut self, src: &[u8]) -> Result<()>
Append src to the input stream. Complete blocks are encoded
immediately; the trailing partial block stays buffered until enough
input arrives or flush / finish
runs. Returns an error if called after finish without an intervening reset.
One stream produces nothing here at all. The block size is capped at
pledged_src_size, so a
pledge that fits in one block – at default parameters, any pledge of
128 KiB or less – is the block size, and the whole content then waits
for finish, so that the single block carries the last-block flag
itself instead of trailing an empty block behind it. A caller pumping
read between pushes sees the frame header and then
nothing until it finishes. This is upstream’s behavior under the same
pledge, and it makes the frame identical to
encode_all_with_options’.
Sourcepub fn flush(&mut self) -> Result<()>
pub fn flush(&mut self) -> Result<()>
Encode any buffered partial block as a non-final block, making its
output bytes available via take_output. Useful
for forcing progress on slow producers.
Sourcepub fn finish(&mut self) -> Result<()>
pub fn finish(&mut self) -> Result<()>
Emit the final block (marked as last) and, if checksums are enabled,
the four-byte content checksum. After this returns the frame is
complete and push/flush will fail until reset runs.
Sourcepub fn reset(&mut self) -> Result<()>
pub fn reset(&mut self) -> Result<()>
Begin a fresh frame on the same context, preserving allocations and
the configured dictionary. Must be called only after finish.
Sourcepub fn take_output(&mut self) -> Vec<u8> ⓘ
pub fn take_output(&mut self) -> Vec<u8> ⓘ
Hand back the bytes produced so far and reset the internal output
buffer to empty. Safe to call at any time; combine with push /
finish to drain output incrementally.
This hands over the buffer itself, so the encoder starts the next block
with no output capacity and grows a fresh allocation for it. That is the
right trade when the caller wants to own the Vec, and the wrong one
for a pump that drains every block into somewhere else:
read and pending_output reuse
the buffer instead and allocate nothing per block.
Sourcepub fn read(&mut self, dst: &mut [u8]) -> usize
pub fn read(&mut self, dst: &mut [u8]) -> usize
Copy up to dst.len() compressed bytes into dst and return how many
were copied. Bytes that are read are removed from the internal buffer,
which keeps its capacity for the next block.
This is the counterpart to
StreamingDecoder::read, and the
closest thing here to handing upstream a ZSTD_outBuffer: a pump built
on it allocates once for dst and never again.
use zstandard::{EncoderOptions, StreamingEncoder, decode_all};
let payload = b"streamed into a fixed buffer".repeat(4_000);
let mut encoder = StreamingEncoder::new(EncoderOptions::default())?;
let mut window = vec![0u8; StreamingEncoder::RECOMMENDED_OUTPUT_SIZE];
let mut compressed = Vec::new();
for chunk in payload.chunks(StreamingEncoder::RECOMMENDED_INPUT_SIZE) {
encoder.push(chunk)?;
loop {
let n = encoder.read(&mut window);
if n == 0 { break; }
compressed.extend_from_slice(&window[..n]);
}
}
encoder.finish()?;
loop {
let n = encoder.read(&mut window);
if n == 0 { break; }
compressed.extend_from_slice(&window[..n]);
}
assert_eq!(decode_all(&compressed)?, payload);Sourcepub fn pending_output(&self) -> &[u8] ⓘ
pub fn pending_output(&self) -> &[u8] ⓘ
Borrow the compressed bytes buffered so far without removing them.
Pair with consume_output to forward output
somewhere that takes a slice, with no copy in between and no allocation
per block. This is what io::Writer uses.
Sourcepub fn consume_output(&mut self, count: usize)
pub fn consume_output(&mut self, count: usize)
Discard the first count bytes of pending_output.
§Panics
If count exceeds pending_output_len.
Consuming more than was produced is a caller bug, and silently clamping
it would drop compressed bytes and leave a truncated frame that still
parses as a frame.
Sourcepub fn pending_output_len(&self) -> usize
pub fn pending_output_len(&self) -> usize
Number of compressed bytes currently waiting to be drained by take_output.
Sourcepub fn is_finished(&self) -> bool
pub fn is_finished(&self) -> bool
true if finish has been called and no reset has
followed it.