Crate iris_format 

The container that carries an iris dataset.

A container is a header, a run of sections, a footer that describes them, and a trailer that says where the footer is. The footer holds the schema, the reference to the decoder that reads this dataset, and a digest for every section. Nothing in here knows anything about Arrow or about WebAssembly; the schema and the decoder module are both carried as opaque bytes with a digest, which is what keeps this crate small enough to read in one sitting and small enough to fuzz seriously.

The layout is written out in docs/FORMAT.md and in the module documentation for layout.

Parsing is the untrusted path

A dataset arrives from somewhere. Reading one must not panic, must not read out of bounds, and must not allocate on the basis of a length field it has not checked. The crate forbids unsafe outright, so out of bounds is a language guarantee rather than a promise. The other two are properties of how the parser is written, and they are held up by tests and by a fuzz target.

The allocation rule is the one that is easiest to get wrong, so the format is arranged to make it hard: there is no count field anywhere in the footer. The number of sections is however many section records the footer actually contains, so a file that claims a billion sections has to be large enough to hold a billion section records.

Digests

Each section carries the digest of its bytes, the footer carries the section records, and the trailer carries a digest over the header and the footer. Container::parse checks the last of those, because it is cheap and it makes a parsed container mean the metadata is what the writer wrote. Container::verify checks the sections, which means reading the whole file, so it is a separate decision that a caller makes once when a dataset arrives.

Re-exports

pub use layout::DecoderLocation;

pub use layout::FORMAT_MAJOR;

pub use layout::FORMAT_MINOR;

pub use layout::MAGIC;

pub use layout::SchemaEncoding;

pub use layout::SectionKind;

Modules

layout

Where everything sits in the file.

Structs

Builder

Builds a container.

Container

A parsed container, borrowing the bytes it was parsed from.

Dataset

What the dataset is and how big it is.

DecoderRef

Which decoder reads this dataset.

Digest

A BLAKE3 hash of some part of a container.

Directory

What a container holds, without the bytes it holds.

FileHeader

The fixed fields at the front of a container.

Placement

Where the footer is, read from the trailer.

Schema

The Arrow schema, carried as bytes this crate does not look inside.

Section

One run of bytes in the payload area.

Enums

Error

Why a container could not be read, or could not be trusted once it was read.

Constants

VERSION

The version of this crate, as reported by build metadata.

Type Aliases

Result

The result of reading a container.