riegeli-rs

A pure-Rust, byte-level compatible implementation of Google's Riegeli/records file format — a high-performance, seekable, compressed record store used in machine learning and data pipelines.

Why Riegeli?

Riegeli files combine fast sequential writes, random-access reads, and transparent compression into a single container format. Records are grouped into chunks, chunks are aligned to 64 KiB blocks with integrity hashes at every boundary, and the whole file can be seeked by numeric position without scanning from the start. This crate brings that format to the Rust ecosystem with no C++ dependency.

Quick start

[dependencies]
riegeli = "0.1"

Write three records and read them back:

use std::io::Cursor;
use riegeli::{RecordWriter, RecordReader, WriterOptions, ReaderOptions, CompressionType};

// Write
let mut buf = Vec::new();
let opts = WriterOptions::new().compression(CompressionType::Zstd);
let mut writer = RecordWriter::new(Cursor::new(&mut buf), opts).unwrap();
writer.write_record(b"alpha").unwrap();
writer.write_record(b"bravo").unwrap();
writer.write_record(b"charlie").unwrap();
writer.close().unwrap();

// Read
let mut reader = RecordReader::new(Cursor::new(&buf), ReaderOptions::new()).unwrap();
while let Some(record) = reader.read_record().unwrap() {
    println!("{}", std::str::from_utf8(&record).unwrap());
}

File format overview

A Riegeli file is divided into fixed-size blocks of 65 536 bytes. Every block boundary carries a 24-byte BlockHeader containing a HighwayHash-64 integrity hash and previous_chunk / next_chunk file-offset pointers (little-endian u64). These pointers allow a reader that lands on any block boundary to locate the nearest chunk without scanning.

The first block header is at offset 0. Immediately after it, at offset 24, sits a 40-byte file-signature chunk (chunk type 's').

Data is stored in chunks. Each chunk begins with a 40-byte ChunkHeader:

Chunk types:

Compression (first byte of chunk data for compressed types):

Public API

RecordWriter

let opts = WriterOptions::new()
    .compression(CompressionType::Zstd)
    .transpose(true)          // enable columnar encoding for proto records
    .chunk_size(1 << 20)      // flush every ~1 MiB of record data
    .initial_padding(65536);  // pad file size to a multiple on close

let mut writer = RecordWriter::new(dest, opts)?;
writer.write_record(b"data")?;
writer.flush()?;   // ensure buffered records are written
writer.close()?;   // finalize the file; further writes return Err

RecordReader

let mut reader = RecordReader::new(source, ReaderOptions::new())?;

// Sequential read
while let Some(record) = reader.read_record()? {
    process(&record);
}

// Position and seek
let pos = reader.last_pos();      // RecordPosition of last-read record
let n = pos.numeric();            // u64 suitable for storage
reader.seek_numeric(n)?;          // return to that record
let same = reader.read_record()?; // re-read it

// Metadata
if let Some(meta) = reader.read_metadata()? {
    // meta is a RecordsMetadata proto
}

// Field projection (transpose chunks only)
use riegeli::{Field, FieldProjection};
let proj = FieldProjection::new()
    .add_field(Field::new(vec![1]))   // include proto field 1
    .add_field(Field::new(vec![2]));  // include proto field 2
let opts = ReaderOptions::new().field_projection(proj);
let mut reader = RecordReader::new(source, opts)?;

RecordPosition

Returned by RecordReader::pos() and last_pos(). Call .numeric() to obtain a u64 suitable for persistence, then pass to seek_numeric to restore position.

Cargo features

To use Brotli and Zstd with no Snappy:

riegeli = { git = "...", features = ["brotli", "zstd"] }

Implementation status

All phases are complete and byte-level compatible with the C++ reference implementation.

Conformance is verified against the C++ reference implementation via an FFI test harness (riegeli-ffi) that calls the C++ reader and writer over a cxx bridge. The test suite uses golden files produced by C++ to validate byte-level interoperability in both directions.

Build requirements

This crate generates Rust code from .proto files at build time using protobuf-codegen, which requires a compatible protoc binary on your PATH. Download the latest release from the protobuf releases page (look for protoc-<version>-<platform>.zip).

Dependencies

Dev dependencies: proptest, criterion.

Benchmarks

See riegeli/benches/README.md for the full head-to-head Rust vs. C++ benchmark matrix. Representative results on Linux x86-64 (10 000 records, large payload):

C++ read throughput is measured through the FFI bridge and includes a per-record copy across the boundary, making it lower than native C++ performance.

License

Apache-2.0