mopenzl

OpenZL compression and decompression in safe Rust.

A port of Meta's OpenZL v0.3.0, the format-aware compression framework, with a Rust-native API. Compress serial bytes with the default graph or typed, multiple inputs with any standard or custom graph; decode every standard codec with one universal decoder; reuse working memory between payloads; integrate with std::io::Read and std::io::Write; and drive either direction incrementally.

Quick start · Benchmarks · Structured data · Read / Write · User guide · Graphs · Compatibility · Changelog

Why mopenzl?

• Safe implementation, not a C wrapper. Production code in this crate forbids unsafe. Zstandard comes from the safe-Rust mzstd; every other codec of the OpenZL graph is ported here. SIMD kernels use fearless_simd.
• Byte-identical encoder. For the same input, graph and global parameters the frames are exactly those of OpenZL v0.3.0, at format versions 21 to 27. See compatibility for the conditions.
• Universal decoder. One decoder reads the frame of every standard codec, including frames with typed and multiple outputs (Decompressor::decompress_typed).
• Graphs, typed inputs and dictionaries. Every standard node and graph is ported (compress_generic, entropy coders, LZ families, the numeric transformer selector, ml_selector, clustering, SDDL and SDDL2, segmenters, brute_force, ...). Build your own with GraphBuilder, including Rust selectors, function graphs and segmenters. Zstandard dictionary bundles work in both directions.
• Sans-I/O sessions. EncoderSession and DecoderSession work only with the buffers you pass in and never perform I/O, so you can drive them from blocking code, async runtimes or your own event loop.
• Reusable state. Keep compressors, decompressors and output buffers across requests instead of rebuilding them for every payload.
• Select only the codecs you need. One Cargo feature per codec, collected in all-codecs; no_std with alloc is supported.

Performance

The only figures recorded so far are in the implementation comparison, measured with Criterion on the serial default graph of the initial port against OpenZL v0.3.0 through openzl-ffi. Every compression case asserts byte-identical frames before timing, so compressed sizes are equal by construction.

Apple M5 Pro · rustc 1.99.1 · release profile · Criterion with 1 s warm-up and 2 s measurement · throughput of input bytes · date not recorded. The Rust compressor reuses its state; the C side creates its context per call, as ZL_compress-style callers do, so the 64-byte case mostly measures context setup.

Zstandard dominates compressible inputs, so those rows mirror mzstd against zstd; incompressible inputs are stored, and there the SIMD checksum and the absence of per-call setup decide. "Rust decompress" decodes into a reused slice; into a Vec the records cases measured 1.76 GiB/s (L1) and 2.48 GiB/s (L6). These are single runs on one machine: no statistical significance is claimed, and the typed and graph paths have no recorded benchmarks yet. Reproduce.

Quick start

Requires Rust 1.89 or later. Add it to your project:

[dependencies]
mopenzl = "0.2"

use mopenzl::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = "openzl ".repeat(1000);
    let config = EncoderConfig::default().with_level(Level::try_from(6)?);
    let mut encoder = Compressor::new(config)?;

    let frame = encoder.compress(input.as_bytes())?;
    println!("{} -> {} bytes", input.len(), frame.len());

    let restored = Decompressor::default().decompress(&frame)?;
    assert_eq!(restored, input.as_bytes());
    Ok(())
}

EncoderConfig::default() is the reference's default: level 6, the newest format version (27) and both checksums on. Levels run from -131072 to 22.

Choose your API

Compression and decompression expose the same core I/O shapes. Pick the shape that matches how your application already moves bytes.

Reuse memory between payloads

Compressor and Decompressor own reusable working state. Keep the codec and your output buffer alive across operations when allocation reuse matters.

use mopenzl::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(3)?))?;
    let mut decoder = Decompressor::default();
    let mut frame = Vec::new();

    for input in [b"first payload".as_slice(), b"second payload".as_slice()] {
        frame.clear(); // Keep the allocation; compress_into appends.
        let written = encoder.compress_into(input, &mut frame)?;
        assert_eq!(written, 0..frame.len());
        assert_eq!(decoder.decompress(&frame)?, input);
    }
    Ok(())
}

RetentionPolicy controls what a codec keeps between operations; see the user guide.

Read and Write streaming

Both codecs provide synchronous adapters for the standard Rust I/O traits:

• reader(...) wraps an input Read and exposes transformed bytes through Read.
• writer(...) wraps an output Write and accepts source bytes through Write.

A frame header records the frame's total size, so declare it with InputSize::Exact when you know it: a session then writes the header at once and, with Chunking::Fixed, holds at most one chunk of input. With InputSize::Unknown it holds the whole input until the frame ends. Encoder writers must be explicitly finished; dropping one abandons the frame.

use mopenzl::{ChunkSize, Chunking, Compressor, Decompressor, EncoderConfig, InputSize};
use std::io::{Read, Write};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let payload = b"streamed payload ".repeat(10_000);
    let config = EncoderConfig::default().with_chunking(Chunking::Fixed(ChunkSize::MIN));

    // Push model: write plain bytes into a compressed sink.
    let mut encoder = Compressor::new(config)?;
    let mut writer = encoder.writer(Vec::new(), InputSize::Exact(payload.len() as u64).into());
    writer.write_all(&payload)?;
    let frame = writer
        .finish()
        .map_err(mopenzl::io::FinishError::into_error)?;

    // Pull model: read compressed bytes from a plain source.
    let mut reader = encoder.reader(&payload[..], InputSize::Exact(payload.len() as u64).into());
    let mut from_reader = Vec::new();
    reader.read_to_end(&mut from_reader)?;
    assert_eq!(from_reader, frame);

    // Decompression, pulling restored bytes from a compressed source.
    let mut decoder = Decompressor::default();
    let mut restored = Vec::new();
    decoder.reader(&frame[..]).read_to_end(&mut restored)?;
    assert_eq!(restored, payload);

    // ... or pushing compressed bytes into a restored sink.
    let mut writer = decoder.writer(Vec::new());
    writer.write_all(&frame)?;
    let restored = writer
        .finish()
        .map_err(mopenzl::io::FinishError::into_error)?;
    assert_eq!(restored, payload);
    Ok(())
}

The decoder adapters use bounded internal buffering, and decoder-writer finalization reports truncated or invalid input instead of silently accepting an incomplete frame. See streaming and completion.

Compress structured data

OpenZL's strength is typed input: integers, fixed-width records and strings are transformed by codecs that know their shape before an entropy coder sees them. Wrap borrowed data in mopenzl::input types, pick a standard graph, and decode with decompress_typed.

use mopenzl::graph::{CompressorGraph, StandardGraph};
use mopenzl::input::{Input, Numbers};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 10 000 slowly growing 32-bit timestamps, little-endian.
    let values: Vec<u8> = (0..10_000u32)
        .flat_map(|i| (1_700_000_000 + i * 3).to_le_bytes())
        .collect();
    let numbers = Numbers::try_from((&values[..], 4))?;

    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(CompressorGraph::standard(StandardGraph::SelectNumeric))
        .build()?;
    let frame = compressor.compress_inputs(&[Input::from(numbers)])?;
    println!("{} -> {} bytes", values.len(), frame.len());

    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs[0].data_type(), DataType::Numeric);
    assert_eq!(outputs[0].width(), 4);
    assert_eq!(outputs[0].content(), values);
    Ok(())
}

Compressor::compress and the byte-oriented decoder methods handle one serial input and one serial output, like ZL_compress and ZL_decompress; frames with typed or several outputs go through decompress_typed. The structured data guide covers numeric, record and string inputs, clustering, ML selectors and SDDL; the graph guide lists every standard graph.

Build your own graph

A frame records the codecs that produced it, not the graph, so any graph decodes with the same universal decoder. GraphBuilder derives new graphs from the standard ones: pipelines of nodes, parameterized copies, and selectors, function graphs and segmenters written in Rust.

use mopenzl::graph::{GraphBuilder, StandardGraph, StandardNode};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Read serial bytes as 32-bit integers, take their deltas, then field-LZ them.
    let mut builder = GraphBuilder::new();
    let start = builder.pipeline(
        &[
            StandardNode::ConvertSerialToNumLe32.into(),
            StandardNode::DeltaInt.into(),
        ],
        StandardGraph::FieldLz,
    )?;
    let graph = builder.build(start)?;

    let input: Vec<u8> = (0..4096u32).flat_map(|i| (i * 7).to_le_bytes()).collect();
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress(&input)?;
    assert_eq!(Decompressor::default().decompress(&frame)?, input);
    Ok(())
}

Selectors and segmenters, local parameters, chunking and permissive fallback are in the graph guide.

Dictionaries

A Zstandard dictionary travels in a DictBundle; the frame records the bundle's ID and the decoder resolves it through a BundleStore. Parameterize the Zstandard node with the dictionary, load the bundle, then build the graph.

use mopenzl::dict::{BundleStore, Dict, DictBundle};
use mopenzl::graph::{GraphBuilder, NodeParams, StandardGraph, StandardNode};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Training data: 40 records of the shape we will compress.
    let record = |i: usize| {
        let users = ["alice", "bob", "carol", "dave", "erin"];
        let cities = ["Oslo", "Lima", "Kyiv", "Pune"];
        format!(
            r#"{{"user":"{}","city":"{}","plan":"{}","seen":{},"flags":["beta","email"],"locale":"en-GB"}}"#,
            users[i % 5],
            cities[i % 4],
            if i.is_multiple_of(2) { "pro" } else { "free" },
            i * 7
        )
    };
    let history = (0..40).map(record).collect::<Vec<_>>().join("\n");
    let dict = Dict::zstd(history.as_bytes(), 3)?;
    let bundle = DictBundle::try_from(vec![dict.clone()])?;

    let mut builder = GraphBuilder::new();
    let node = builder.parameterize_node_with(
        StandardNode::Zstd,
        NodeParams::default().with_dictionary(dict.id()),
    )?;
    builder.load_dictionaries(&bundle)?;
    let start = builder.static_graph(node, &[StandardGraph::Store.into()])?;
    let graph = builder.build(start)?;

    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let payload = record(1234).into_bytes();
    let frame = compressor.compress(&payload)?;
    let plain = Compressor::default().compress(&payload)?;
    println!(
        "{} bytes -> {} with the dictionary, {} without",
        payload.len(),
        frame.len(),
        plain.len()
    );

    let mut store = BundleStore::default();
    store.insert(&bundle)?;
    let mut decoder = Decompressor::builder(Default::default())
        .with_dictionaries(store)
        .build();
    assert_eq!(decoder.decompress(&frame)?, payload);
    Ok(())
}

Dictionaries need format version 25 or newer. See the dictionary guide.

Select only the codecs you need

The default feature set is std, compression, decompression and all-codecs. all-codecs enables one Cargo feature per OpenZL codec (zstd, entropy, lz4, field-lz, delta, tokenize, ...; the codec guide lists them all). Disable default features to select directions and codecs yourself:

[dependencies]
mopenzl = { version = "0.2", default-features = false, features = ["std", "decompression", "zstd"] }

Without a codec's feature the encoder never writes it (a chunk that would use it is stored, so frames stop matching the reference's default graph) and the decoder reports frames that use it as DecodeError::Unsupported. Frames of the default graph need only zstd.

For no_std with alloc, disable default features and enable no_std:

[dependencies]
mopenzl = { version = "0.2", default-features = false, features = ["no_std", "decompression", "all-codecs"] }

no_std requires a global allocator, removes the std::io adapters and runtime SIMD detection (the compile-time baseline is used), and pulls in libm for the estimators the numeric selectors rank candidates with. Caveat: libm's expf differs from the platform's by one ULP for some inputs, so the numeric transformer selector's scores can differ slightly from the reference's; decisions change only on exact ties (none were observed in 4000+ frames), but byte identity of encoder decisions that depend on floating point is not guaranteed under no_std. Cargo features are additive: another dependency can re-enable std.

Compatibility

The reference in openzl-ffi/vendor is pinned to OpenZL v0.3.0 with zstd 1.5.7 and lz4 1.10.0.

• Encoder. For the same input, graph (or the serial segmenter, with Chunking::Fixed), level, format version and checksum settings, Compressor writes the frame ZL_CCtx_compress writes, at format versions 21 to 27 and levels -131072..=22. User-written selectors, function graphs and segmenters are byte-identical when they make the same decisions as the reference's counterpart. The no_std caveat above applies.
• Decoder. Reads every standard codec valid at format versions 21 to 27, with both checksums verified. On malformed data it deviates from the reference only where the reference reads uninitialized or out-of-bounds memory or wraps silently (offset 0 in LZ codecs, out-of-range partition bucket IDs or flatpack indices, over-reads in the struct Huffman decoder, reads past the extra bits of quantize codes, overflowing sizes): this decoder reports an error. See decoder codecs.
• Not supported. Frames of format versions below 21 and legacy codecs that only exist there (rolz, fastlz, zstd_fixed, the deprecated FSE and Huffman codecs), custom codecs registered by another application (Unsupported), and training.
• Dictionaries. Only Zstandard dictionaries can be materialized.

Typed and multiple inputs, the graph API and dictionaries follow the reference's semantics; mopenzl does not provide a C ABI, so it is not a drop-in replacement for libopenzl.

Validation

The repository includes differential tests against the pinned C library (frames byte-compared for every format version 21 to 27 and for each standard node and graph, decoder agreement on C-produced frames), cross-API and cross-backend checks, AFL++ and libFuzzer targets, Miri and AddressSanitizer runs over the SIMD and uninitialized-output paths, and a 100% function coverage gate. See compatibility and fuzzing for what is covered and how to reproduce it.

The crate enforces #![cfg_attr(not(test), forbid(unsafe_code))]. This describes mopenzl's production implementation, not every transitive dependency: SIMD intrinsics are encapsulated by fearless_simd, and differential tests use a C FFI reference.

Documentation and contributing

API reference · User guide · Graphs · Codecs · Structured data · Dictionaries · Architecture

Bug reports with a minimal input, graph, parameters and reproduction steps are welcome. For performance reports, include the CPU, compiler, feature set and workload. If compression output differs from OpenZL v0.3.0 for the same graph and parameters, open an issue with both frames.

Read CONTRIBUTING.md for the checks a change must pass, and follow the Code of Conduct. The development guide covers local checks, coverage and fuzzing. Report security vulnerabilities privately as described in SECURITY.md, not as public issues.

License

MIT or APACHE 2.0 at your option. mopenzl is derived from OpenZL, which is BSD-licensed; see NOTICE for its attribution.