Burn Pack

The burnpack binary serialization format for the Burn deep learning framework

burn-pack reads and writes the burnpack container format. It is tensor-library-agnostic and dependency-light: it depends only on burn-std (for DType / Bytes), serde, and a CBOR codec — it knows the on-disk format but has no notion of Burn modules. Tensor data is Bytes-native and read lazily from files (256-byte aligned for zero-copy mmap and efficient GPU transfers), and the reader is hardened against malformed input.

If you just want to save and load Burn models, use the higher-level burn-core record API or burn-store (which adds PyTorch/SafeTensors interop). Reach for burn-pack directly only to produce or consume the raw format.

Usage

use burn_pack::{Bytes, DType, Reader, Tensor, Writer};

let raw: Vec<u8> = [1.0f32, 2.0, 3.0, 4.0].iter().flat_map(|v| v.to_le_bytes()).collect();
let tensor = Tensor::new("weight".to_string(), DType::F32, vec![2, 2], None, Bytes::from_bytes_vec(raw));

let packed = Writer::new(vec![tensor]).into_bytes().unwrap();
let reader = Reader::from_bytes(packed).unwrap();
assert_eq!(reader.into_tensors().unwrap()[0].shape.to_vec(), vec![2, 2]);

Use Writer::write_to_file / Reader::from_file for disk I/O (the default std feature; disable it for no-std targets). write_to_file builds the container beside the destination and renames it into place once complete, so the old file survives a save that fails partway, whether from a full disk or a deferred tensor whose provider errors mid-write. write_to_file_in_place truncates and rewrites the destination instead: no fsync and no transient second copy, but a failed save leaves it truncated. Writer::overwrite(false) makes both refuse an existing destination, checked when the file is created or published rather than beforehand. See the API docs for how far each guarantee reaches.

Writing models larger than memory

A tensor's bytes need not exist yet. Tensor::deferred describes one by its length plus a provider that yields the data on demand, so the writer can lay out the whole container before any I/O and then draw the bytes one tensor at a time, dropping each before asking for the next:

let tensor = Tensor::deferred(
    name,
    DType::F32,
    shape,
    None,
    rows * cols * 4,              // length, without the data
    move || weight.read_from_device(),  // called once, when the writer reaches it
);

Paired with write_to_file, peak host memory is bounded by the largest single tensor rather than by the whole model. burn-store collects a module's parameters directly as these, so saving a Module gets it for free.

See the docs for the format layout and the full API.

License

This project is dual-licensed under MIT and Apache-2.0.

Part of the Burn deep learning framework. See the Burn Book and the API documentation.