iris_format/
lib.rs

1//! The container that carries an iris dataset.
2//!
3//! A container is a header, a run of sections, a footer that describes them, and a trailer that
4//! says where the footer is. The footer holds the schema, the reference to the decoder that reads
5//! this dataset, and a digest for every section. Nothing in here knows anything about Arrow or
6//! about `WebAssembly`; the schema and the decoder module are both carried as opaque bytes with a
7//! digest, which is what keeps this crate small enough to read in one sitting and small enough to
8//! fuzz seriously.
9//!
10//! The layout is written out in `docs/FORMAT.md` and in the module documentation for [`layout`].
11//!
12//! # Parsing is the untrusted path
13//!
14//! A dataset arrives from somewhere. Reading one must not panic, must not read out of bounds, and
15//! must not allocate on the basis of a length field it has not checked. The crate forbids `unsafe`
16//! outright, so out of bounds is a language guarantee rather than a promise. The other two are
17//! properties of how the parser is written, and they are held up by tests and by a fuzz target.
18//!
19//! The allocation rule is the one that is easiest to get wrong, so the format is arranged to make
20//! it hard: there is no count field anywhere in the footer. The number of sections is however many
21//! section records the footer actually contains, so a file that claims a billion sections has to be
22//! large enough to hold a billion section records.
23//!
24//! # Digests
25//!
26//! Each section carries the digest of its bytes, the footer carries the section records, and the
27//! trailer carries a digest over the header and the footer. [`Container::parse`] checks the last of
28//! those, because it is cheap and it makes a parsed container mean the metadata is what the writer
29//! wrote. [`Container::verify`] checks the sections, which means reading the whole file, so it is a
30//! separate decision that a caller makes once when a dataset arrives.
31
32#![forbid(unsafe_code)]
33
34mod build;
35mod container;
36mod digest;
37mod error;
38pub mod layout;
39mod meta;
40
41pub use build::Builder;
42pub use container::{Container, FileHeader};
43pub use digest::Digest;
44pub use error::{Error, Result};
45pub use layout::{DecoderLocation, FORMAT_MAJOR, FORMAT_MINOR, MAGIC, SchemaEncoding, SectionKind};
46pub use meta::{Dataset, DecoderRef, Schema, Section};
47
48/// The version of this crate, as reported by build metadata.
49pub const VERSION: &str = env!("CARGO_PKG_VERSION");
iris_format/lib.rs

iris_format/
lib.rs
