Struct FrameCompressor 

pub struct FrameCompressor<R: Read = &'static [u8], W: Write = Vec<u8>, M: Matcher = MatchGeneratorDriver, D: DictionaryHold = EncoderDictionary> { /* private fields */ }

An interface for compressing arbitrary data with the ZStandard compression algorithm.

FrameCompressor will generally be used by:

1. Initializing a compressor by providing a buffer of data using FrameCompressor::new()
2. Starting compression and writing that compression into a vec using FrameCompressor::begin

Examples

use structured_zstd::encoding::{FrameCompressor, CompressionLevel};
let mock_data: &[_] = &[0x1, 0x2, 0x3, 0x4];
let mut output = std::vec::Vec::new();
// Initialize a compressor.
let mut compressor = FrameCompressor::new(CompressionLevel::Uncompressed);
compressor.set_source(mock_data);
compressor.set_drain(&mut output);

// `compress` writes the compressed output into the provided buffer.
compressor.compress();

Implementations

impl<R: Read, W: Write> FrameCompressor<R, W, MatchGeneratorDriver>

pub fn new(compression_level: CompressionLevel) -> Self

Create a new FrameCompressor

pub fn set_sequence_producer( &mut self, producer: Option<Box<dyn SequenceProducer>>, fallback: bool, )

Ask producer for each block’s parse in place of the built-in matchers; None removes it. With fallback a block the producer fails on is matched as usual; without, it goes out uncompressed and Self::take_sequence_producer_failure reports it. With long-distance matching on, the long matches are taken outright and the producer parses the literals between them.

pub fn take_sequence_producer_failure(&mut self) -> bool

Whether a block went out uncompressed because the producer failed on it with no fallback, since this was last asked.

pub fn set_parameters(&mut self, params: &CompressionParameters)

Configure fine-grained compression parameters.

Resets the base CompressionLevel to the parameters’ level and installs the per-knob overrides (window/hash/chain/search logs, strategy, LDM) applied at the next frame. Pass None-equivalent (a builder that overrides nothing) to fall back to plain level-based compression.

use structured_zstd::encoding::{
    CompressionLevel, CompressionParameters, FrameCompressor, Strategy,
};
let params = CompressionParameters::builder(CompressionLevel::Level(19))
    .strategy(Strategy::Btultra2)
    .enable_long_distance_matching(true)
    .build()
    .unwrap();
let mut compressor: FrameCompressor = FrameCompressor::new(CompressionLevel::Default);
compressor.set_parameters(&params);
let compressed = compressor.compress_independent_frame(b"some data to compress");
assert!(!compressed.is_empty());

pub fn compress_independent_frame_into( &mut self, input: &[u8], out: &mut Vec<u8>, )

Compress one contiguous &[u8] as a single independent Zstd frame, writing the frame bytes into out (its previous contents are replaced and its allocation reused), reusing this compressor’s heavy state across calls.

This is the reusable-compression-context (CCtx-equivalent) entry point, mirroring C ZSTD_compress2 over a reused ZSTD_CCtx: construct ONE FrameCompressor and call this in a loop to emit N independent, self-describing frames (each carrying its own header, blocks, and checksum, decodable in isolation, with no cross-frame match history). Every call resets the per-frame state: only the allocations are kept, so the dominant per-frame setup cost (table allocation + dictionary prime) is paid once instead of N times. Passing the same out buffer each call additionally reuses the output allocation, matching C’s caller-owned dst buffer (no per-frame output allocation).

Reusing the context + out across many small frames (the typical per-block-frame workload) is far cheaper than a fresh compress_slice_to_vec per block, which allocates and primes from scratch each time.

The input is read in place: no Self::set_source / Self::set_drain setup is required, and the input lifetime is not baked into the compressor type, so successive calls may pass slices with unrelated lifetimes. When the Fast (Simple) backend is active and no dictionary is set, the matcher references the input directly (no per-block history copy); other backends / dictionary use copy each block into history exactly as the streaming compress path does. The source-size hint is derived from the input length on every call, so per-frame table sizing tracks each frame’s actual size regardless of any earlier hint.

A sticky dictionary set via set_dictionary (or its variants) is primed into every frame, mirroring ZSTD_CCtx_loadDictionary / ZSTD_CCtx_refCDict.

Panics

Panics on encoder error, matching Self::compress and compress_slice_to_vec.

pub fn compress_independent_frame(&mut self, input: &[u8]) -> Vec<u8>

Convenience wrapper over Self::compress_independent_frame_into that allocates and returns a fresh Vec per call. Prefer the _into form in tight per-block-frame loops to reuse one output buffer across frames (the CCtx-equivalent zero-per-call-alloc output, matching C’s caller-owned dst).

use structured_zstd::encoding::{FrameCompressor, CompressionLevel};
let mut cctx: FrameCompressor = FrameCompressor::new(CompressionLevel::Default);
let frame_a = cctx.compress_independent_frame(b"first block payload");
let frame_b = cctx.compress_independent_frame(b"second block payload");
assert!(!frame_a.is_empty() && !frame_b.is_empty());

impl<R: Read, W: Write, M: Matcher> FrameCompressor<R, W, M>

pub fn new_with_matcher(matcher: M, compression_level: CompressionLevel) -> Self

Create a new FrameCompressor with a custom matching algorithm implementation

pub fn set_dictionary( &mut self, dictionary: Dictionary, ) -> Result<Option<EncoderDictionary>, DictionaryDecodeError>

Attach a pre-parsed dictionary to be used for subsequent compressions.

In compressed modes, the dictionary id is written only when the active matcher supports dictionary priming. Uncompressed mode and non-priming matchers ignore the attached dictionary at encode time.

pub fn set_dictionary_from_bytes( &mut self, raw_dictionary: &[u8], ) -> Result<Option<EncoderDictionary>, DictionaryDecodeError>

Parse and attach a dictionary blob, in either of the two forms upstream ZSTD_CCtx_loadDictionary takes (ZSTD_dct_auto): a serialized dictionary, and anything else is raw content, which is why any file can be handed to zstd -D. Raw content has no id, so the frame records none and the decoder must be given the same bytes explicitly.

A serialized blob parses through the encoder-only path (skips the FSE/HUF decode lookup-table build the encoder never reads); the entropy ENCODER tables — and thus the emitted frame — are identical to a full parse. To reject anything but a serialized dictionary, parse with EncoderDictionary::from_bytes (upstream ZSTD_dct_fullDict) and attach the result.

An empty buffer is how the same upstream entry point is told there is no dictionary: it clears whatever was attached and succeeds, returning it, rather than reporting a dictionary too small to use.

impl<R: Read, W: Write, M: Matcher, D: DictionaryHold> FrameCompressor<R, W, M, D>

pub fn new_with_matcher_and_dictionary( matcher: M, compression_level: CompressionLevel, dictionary: Option<D>, ) -> Result<Self, DictionaryDecodeError>

A compressor with a custom matcher that compresses every frame with dictionary, or with none, held as given: a &EncoderDictionary is borrowed for the compressor’s lifetime, so a compressor built for one frame reads a dictionary prepared once without taking a reference on it.

Errors

As Self::set_encoder_dictionary.

Examples

use structured_zstd::encoding::sequences::{ExternalSequence, SequenceReplay};
use structured_zstd::encoding::{CompressionLevel, EncoderDictionary, FrameCompressor};

let dictionary = EncoderDictionary::from_serialized_or_raw_content(b"shared history").unwrap();
let input = b"shared history, once more";
// One block of literals; a parse may also reach into the dictionary.
let parse = [ExternalSequence::delimiter(input.len() as u32)];
let mut frame = Vec::new();
let mut compressor: FrameCompressor<&[u8], &mut Vec<u8>, _, _> =
    FrameCompressor::new_with_matcher_and_dictionary(
        SequenceReplay::new(&parse, 1 << 17).with_dictionary(),
        CompressionLevel::Default,
        Some(&dictionary),
    )
    .unwrap();
compressor.set_source(&input[..]);
compressor.set_drain(&mut frame);
compressor.compress();
assert!(compressor.dictionary().is_some());
drop(compressor);
assert!(!frame.is_empty());

pub fn set_magicless(&mut self, magicless: bool)

Enable or disable magicless frame format (ZSTD_f_zstd1_magicless).

When set to true, emitted frames omit the 4-byte magic number prefix. The matching decoder must be configured to expect a magicless stream — wire-format only round-trips with a magicless-aware decoder.

pub fn set_content_checksum(&mut self, emit: bool)

Enable or disable the trailing XXH64 content checksum (semantics of upstream ZSTD_c_checksumFlag). Default false, matching the upstream library default (ZSTD_c_checksumFlag = 0) so out-of-the-box frames carry the same layout and pay the same costs as the reference implementation.

When false, emitted frames set Content_Checksum_flag = 0 and carry no trailing digest; such frames are valid (RFC 8878) and decode correctly in any ContentChecksum mode. Without the hash feature no checksum is emitted regardless of this setting.

pub fn set_content_size_flag(&mut self, emit: bool)

Enable or disable recording Frame_Content_Size in the frame header when the total size is known (semantics of upstream ZSTD_c_contentSizeFlag). Default true, matching upstream. With the flag off the header carries a window descriptor instead (and the single-segment layout, which requires an FCS, is disabled).

pub fn set_dictionary_id_flag(&mut self, emit: bool)

Enable or disable recording the dictionary ID in the frame header when a dictionary is attached (semantics of upstream ZSTD_c_dictIDFlag). Default true, matching upstream. Frames emitted with the flag off still decode when the decoder is handed the dictionary explicitly.

pub fn set_target_block_size(&mut self, target: Option<u32>)

Set an upper bound on emitted block sizes (semantics of upstream ZSTD_c_targetCBlockSize): every physical block’s payload is capped at target bytes (+3-byte block header on the wire), trading some ratio for bounded per-block latency. The value is clamped to [MIN_TARGET_BLOCK_SIZE, MAX_BLOCK_SIZE] (the upstream bounds). None removes the target.

pub fn set_max_block_size(&mut self, size: Option<u32>)

Cap every block of the frame at size bytes (semantics of upstream ZSTD_c_maxBlockSize), for a decoder that holds only that much of a block at a time. Clamped to [MIN_MAX_BLOCK_SIZE, MAX_BLOCK_SIZE]; None restores the format’s 128 KiB. Combines with Self::set_target_block_size: the smaller of the two applies.

use structured_zstd::encoding::{CompressionLevel, FrameCompressor};

let mut compressor = FrameCompressor::new(CompressionLevel::Default);
compressor.set_max_block_size(Some(1024));
let mut frame = Vec::new();
compressor.set_source(&[7u8; 4096][..]);
compressor.set_drain(&mut frame);
compressor.compress();

pub fn set_sequence_capture(&mut self, capture: bool)

Record the parse every block takes, from the next one on, until turned off: its sequences with plain offsets, each block closed by a delimiter carrying its trailing literals, and a block written out unsearched as a delimiter alone (C ZSTD_generateSequences). An empty input records nothing, as upstream’s does. The frames themselves are unchanged.

use structured_zstd::encoding::{CompressionLevel, FrameCompressor};

let input = b"capture capture capture capture".repeat(10);
let mut compressor: FrameCompressor = FrameCompressor::new(CompressionLevel::Default);
compressor.set_sequence_capture(true);
compressor.compress_independent_frame(&input);
let parse = compressor.take_captured_sequences();
let covered: usize = parse.iter().map(|s| (s.lit_len + s.match_len) as usize).sum();
assert_eq!(covered, input.len());

pub fn take_captured_sequences(&mut self) -> Vec<ExternalSequence>

The parse recorded since Self::set_sequence_capture turned it on, or since it was last taken; empty when nothing records it.

pub fn set_source(&mut self, uncompressed_data: R) -> Option<R>

Before calling FrameCompressor::compress you need to set the source.

This is the data that is compressed and written into the drain.

pub fn set_drain(&mut self, compressed_data: W) -> Option<W>

Before calling FrameCompressor::compress you need to set the drain.

As the compressor compresses data, the drain serves as a place for the output to be writte.

pub fn set_source_size_hint(&mut self, size: u64)

Provide a hint about the total uncompressed size for the next frame.

When set, the encoder selects smaller hash tables and windows for small inputs, matching the C zstd source-size-class behavior.

This hint applies only to frame payload bytes (size). Dictionary history is primed separately and does not inflate the hinted size or advertised frame window. Must be called before compress.

pub fn heap_size(&self) -> usize

Total heap bytes this compressor’s allocations hold, excluding the inline struct: the match-finder tables / history / recycled buffers and the primed-dictionary snapshot (via the matcher), the retained Huffman tables (active + recycled spare), the retained dictionary content, the cached dictionary entropy tables (literals Huffman + LL/ML/OF FSE), and the per-block sidecar buffers. Lets a context report its true footprint through ZSTD_sizeof_CCtx.

pub fn compress(&mut self)

Compress the uncompressed data from the provided source as one Zstd frame and write it to the provided drain

This will repeatedly call Read::read on the source to fill up blocks until the source returns 0 on the read call. All compressed blocks are buffered in memory so that the frame header can include the Frame_Content_Size field (which requires knowing the total uncompressed size). The entire frame — header, blocks, and optional checksum — is then written to the drain at the end. This means peak memory usage is O(compressed_size).

To avoid endlessly encoding from a potentially endless source (like a network socket) you can use the Read::take function.

pub fn last_frame_emit_info(&self) -> Option<&FrameEmitInfo>

Available on crate feature lsm only.

Layout of the most recently emitted frame.

Returns None if compress has not been called yet on this compressor. After a successful compress() the returned FrameEmitInfo describes the frame header range, every emitted block’s offset / size / type, and the optional trailing content-checksum range — all in frame-absolute byte offsets matching the bytes written to the drain.

Behind the lsm Cargo feature.

pub fn enable_per_block_checksums(&mut self)

Available on crate features hash and lsm only.

Opt in to per-block XXH64 checksum computation during compress. Default off; zero cost when disabled. The captured digests are accessible via last_frame_block_checksums.

One checksum is emitted per physical FrameBlock written to the drain: 1:1 cardinality with last_frame_emit_info’s blocks vector. On the post-split optimization path (Level 16-22 with large window) the per-partition decompressed range is hashed inside the partition loop so the digest count still matches the emitted block count. The decoder collects per-physical-block digests on the same granularity, so element-wise equality holds round-trip.

Behind all(feature = "lsm", feature = "hash") — the XXH64 primitive lives behind the hash feature, so this method only compiles when both are enabled.

pub fn last_frame_block_checksums(&self) -> Option<&[u32]>

Available on crate features hash and lsm only.

Per-block XXH64 (low 32 bits) digests captured during the most recent compress() call. None unless enable_per_block_checksums was called before compress().

Behind all(feature = "lsm", feature = "hash").

pub fn source_mut(&mut self) -> Option<&mut R>

Get a mutable reference to the source

pub fn drain_mut(&mut self) -> Option<&mut W>

Get a mutable reference to the drain

pub fn source(&self) -> Option<&R>

Get a reference to the source

pub fn drain(&self) -> Option<&W>

Get a reference to the drain

pub fn take_source(&mut self) -> Option<R>

Retrieve the source

pub fn take_drain(&mut self) -> Option<W>

Retrieve the drain

pub fn replace_matcher(&mut self, match_generator: M) -> M

Before calling FrameCompressor::compress you can replace the matcher

pub fn set_compression_level( &mut self, compression_level: CompressionLevel, ) -> CompressionLevel

Before calling FrameCompressor::compress you can replace the compression level.

This also clears any fine-grained parameter overrides installed via set_parameters: reverting to a bare level means plain level-based tuning, not the previous frame’s customized strategy / LDM / log / literal-mode overrides. To keep overriding, call set_parameters again with the new base level.

pub fn compression_level(&self) -> CompressionLevel

Get the current compression level

pub fn set_encoder_dictionary( &mut self, dictionary: D, ) -> Result<Option<D>, DictionaryDecodeError>

Attach an already-parsed EncoderDictionary without reparsing a raw blob, held the way this compressor holds one (see DictionaryHold).

Accepts an EncoderDictionary produced once via EncoderDictionary::from_bytes / EncoderDictionary::from_dictionary or handed back by Self::clear_dictionary / the set_dictionary* return value, so callers can reattach or reuse a prepared dictionary across compressions without re-running the dictionary parse each time. Returns the previously-attached dictionary, if any.

Errors

ZeroRepeatOffsetInDictionary when a repeat offset of the dictionary is zero.

pub fn dictionary(&self) -> Option<&EncoderDictionary>

The dictionary frames are compressed with, if one is attached.

Examples

use structured_zstd::encoding::{CompressionLevel, EncoderDictionary, FrameCompressor};

let dictionary = EncoderDictionary::from_serialized_or_raw_content(b"some shared history").unwrap();
let mut compressor: FrameCompressor = FrameCompressor::new(CompressionLevel::Default);
assert!(compressor.dictionary().is_none());
compressor.set_encoder_dictionary(dictionary).unwrap();
assert_eq!(compressor.dictionary().map(EncoderDictionary::id), Some(0));

pub fn clear_dictionary(&mut self) -> Option<D>

Remove the attached dictionary, returning it as it was held.