Crate mopenzl 

Safe Rust OpenZL codecs, independently selected with Cargo features.

OpenZL is Meta’s format-aware compression framework: a frame records the graph of codecs that produced it, so one universal decoder reads every frame. This crate ports the framework’s core to Rust, starting with what ZL_compress and ZL_decompress do for serial data:

• Byte-identical encoder. For the same input and global parameters (level, format version, checksums), Compressor writes exactly the frame of OpenZL v0.3.0’s ZL_CCtx_compress with its default graph, or, with Chunking::Fixed, of its serial segmenter. Inputs below ten bytes are stored, larger ones go through Zstandard (the safe-Rust mzstd, byte-identical to the zstd 1.5.7 OpenZL vendors), and any chunk that would grow is stored instead.
• Caller-scheduled parallel compression through compressor::parallel (without no_std): one compatible OpenZL frame from independent segments. Byte identity with the reference serial encoder is not guaranteed. Streaming retains compressed chunks until the total input size is known.
• Decoder for frames of format versions 21 to 27 built from those codecs, with both checksums verified, bounded input and output, and precise errors for frames that need codecs not ported yet.
• No unsafe. Kernels use fearless_simd with the instruction set detected once per codec (Backend); every backend writes the same bytes.
• Sans-I/O sessions that work only with the buffers you pass in, plus std::io adapters.
• Indexed resource containers behind the opt-in framed feature: independent graphs per resource, typed outputs, metadata, native sessions and whole-resource seek. Each resource contains one ordinary OpenZL frame; the outer container uses mopenzl’s own format. See framing when enabled.

compression and decompression are enabled by default. Disable default features and select either codec alongside std or no_std:

mopenzl = { version = "0.2", default-features = false, features = ["std", "decompression"] }

Choose your API

I/O shape	Compression	Decompression
Return a new Vec<u8>	Compressor::compress	Decompressor::decompress
Append to an existing Vec	Compressor::compress_into	Decompressor::decompress_into
Write into a caller-owned slice	Compressor::compress_to_slice	Decompressor::decompress_to_slice
Write into uninitialized memory	Compressor::compress_to_uninit	Decompressor::decompress_to_uninit
Choose the graph / codec nodes	Compressor::builder / ParallelCompressor::builder → with_graph	— (the frame carries its own)
Pull output through std::io::Read	Compressor::reader	Decompressor::reader
Push input through std::io::Write	Compressor::writer	Decompressor::writer
Drive input/output incrementally	Compressor::start → EncoderSession	Decompressor::start → DecoderSession
Same, into uninitialized memory	EncoderSession::process_uninit	DecoderSession::process_uninit
Same, with the session owning the codec	Compressor::into_session → EncoderSessionOwned	Decompressor::into_session → DecoderSessionOwned
Split one known input across your threads	prepare_slice / prepare_source → ScopedParallelBatch	—
Stream chunked input across your threads	start → ParallelEncoderSession + ParallelTaskQueue	—
Same, into uninitialized memory	ParallelEncoderSession::process_uninit	—
Same, with the session owning the compressor	into_session → ParallelEncoderSessionOwned	—
Pull parallel output through std::io::Read	reader → ParallelEncoderReader + ParallelTaskQueue	—
Push input through std::io::Write across your threads	writer → ParallelEncoderWriter + ParallelTaskQueue	—

ParallelCompressor lives in compressor::parallel and is unavailable with no_std. The caller runs its tasks; streaming buffers compressed chunks until finish, when the total input size is known.

use mopenzl::{Compressor, Decompressor, EncoderConfig, Level};

let mut encoder = Compressor::new(EncoderConfig::from(Level::try_from(3)?))?;
let mut decoder = Decompressor::default();
let mut frame = Vec::new();
for input in [b"first payload".as_slice(), b"second payload".as_slice()] {
    frame.clear(); // Keep the allocation; compress_into appends.
    encoder.compress_into(input, &mut frame)?;
    assert_eq!(decoder.decompress(&frame)?, input);
}

Streaming and chunks

A frame header records the frame’s total size, and the reference decoder rejects frames that leave it unknown. Declare the size with InputSize::Exact when you know it: a session then writes the header at once and holds at most one chunk of input. With InputSize::Unknown it holds the whole input until the frame ends. Chunking::Fixed bounds a chunk, so with a declared size both the encoder and the decoder work in bounded memory; the default, Chunking::Whole, matches ZL_compress.

use mopenzl::{ChunkSize, Chunking, Compressor, Decompressor, EncoderConfig, InputSize};
use std::io::{Read, Write};

let payload = b"streamed payload ".repeat(10_000);
let config = EncoderConfig::default().with_chunking(Chunking::Fixed(ChunkSize::MIN));
let mut encoder = Compressor::new(config)?;
let mut writer = encoder.writer(Vec::new(), InputSize::Exact(payload.len() as u64).into());
writer.write_all(&payload)?;
let frame = writer.finish().map_err(mopenzl::io::FinishError::into_error)?;

let mut decoder = Decompressor::default();
let mut content = Vec::new();
decoder.reader(&frame[..]).read_to_end(&mut content)?;
assert_eq!(content, payload);

Untrusted input

Limits are unlimited by default; set DecodeLimits for untrusted frames. A frame declaring more output than allowed fails before any chunk is decoded.

use mopenzl::{DecodeLimits, DecoderConfig, Decompressor};

let limits = DecodeLimits::default()
    .with_max_input_bytes(Some(1 << 20))
    .with_max_output_bytes(Some(8 << 20));
let mut decoder = Decompressor::new(limits.into());
assert!(decoder.decompress(b"not a frame").is_err());

Select codecs

Features are additive; the defaults enable everything.

Feature	Enables
compression	Compressor and its sessions
decompression	Decompressor and its sessions
std	the std::io adapters, std::error::Error and runtime SIMD detection
no_std	alloc-only builds with the compile-time SIMD baseline
zstd	the Zstandard codec (mzstd), the default graph’s backend

Every OpenZL codec has its own feature, all enabled by default. Without a codec’s feature the encoder never writes it (chunks that would use it are stored, so frames stop matching the reference’s default graph) and the decoder reports frames using it as Unsupported.

mopenzl = { version = "0.2", default-features = false, features = ["std", "decompression", "zstd"] }

Re-exports

pub use compressor::ChunkSize;

pub use compressor::Chunking;

pub use compressor::Compressor;

pub use compressor::CompressorBuilder;

pub use compressor::ConfigError;

pub use compressor::EncodeError;

pub use compressor::EncoderConfig;

pub use compressor::EncoderSession;

pub use compressor::EncoderSessionOwned;

pub use compressor::EncoderStatus;

pub use compressor::InputSize;

pub use compressor::Level;

pub use compressor::Operation;

pub use compressor::Parameter;

pub use compressor::Progress;

pub use compressor::SizeOverflow;

pub use compressor::StreamConfig;

pub use decompressor::DecodeError;

pub use decompressor::DecodeFailure;

pub use decompressor::DecodeLimits;

pub use decompressor::DecodeOperation;

pub use decompressor::DecodeProgress;

pub use decompressor::DecoderConfig;

pub use decompressor::DecoderSession;

pub use decompressor::DecoderSessionOwned;

pub use decompressor::DecoderStatus;

pub use decompressor::Decompressor;

pub use decompressor::DecompressorBuilder;

pub use decompressor::FrameInfo;

pub use decompressor::FrameMode;

pub use decompressor::InvalidDataKind;

pub use decompressor::TypedOutput;

pub use decompressor::Unsupported;

Modules

compressor

OpenZL compression.

custom_codec

Application-defined OpenZL codecs.

decompressor

OpenZL decompression.

dict

Dictionary bundles (format version 25 and newer).

graph

Compression graphs: standard graphs and nodes, parameterization, and user-defined selectors, function graphs and segmenters.

input

Typed inputs of a frame.

io

Streaming I/O adapters for compression and decompression.

Structs

Backend

A supported execution backend: the SIMD instruction level the codecs’ kernels run with.

FormatVersion

A version of the OpenZL frame format.

UnsupportedFormatVersion

A format version this crate does not support.

Enums

DataType

What an OpenZL input or output holds.

RetentionPolicy

What a codec keeps allocated between operations.