Crate yo_format 

The byte layouts of a .yo file, and nothing else.

This crate is the written form of 07-yo-file-format.md. It knows how to turn each structure into bytes and how to read it back, and it knows nothing about files, memory, shards or the engine. That separation is deliberate: the engine and yodb check both encode with this crate, and the independent minimal reader deliberately does not, so a change here that is not also a change to the specification shows up as the two disagreeing.

Rules that hold everywhere below.

Little endian, unconditionally. A big endian machine byte swaps on the way in and on the way out, and pays for it, because a format whose byte order depends on the writer is not a format.

Every checksum is CRC32C over the bytes named, with the checksum field itself read as zero. That way the check is the same computation whether you are about to write the bytes or have just read them.

No allocation. Everything encodes into a caller supplied slice and decodes from a borrowed one. The engine calls these from inside the shard loop and the shard loop does not allocate.

Re-exports

pub use catalog::Band;

pub use catalog::CatalogEntry;

pub use catalog::Model;

pub use catalog::ValueType;

pub use page::PAGE_HEADER_LEN;

pub use page::PageHeader;

pub use record::RecordHeader;

pub use record::RecordKind;

pub use record::RecordRef;

pub use record::record_flags;

pub use superblock::CheckpointEntry;

pub use superblock::Superblock;

pub use superblock::superblock_flags;

Modules

catalog

The collection catalogue, which is what makes a .yo file self describing.

page

The log page header.

record

The record, which is the unit of everything: a commit, a replay step, and a thing compaction either copies or drops.

superblock

The superblock, its two slots, the run length encoded shard table and the per shard checkpoint entries.

Constants

DATA_START

Where the data starts, which is after both superblock slots.

DEFAULT_PAGE_SIZE

The default segment size, and the size a file gets if nobody chooses.

FORMAT_VERSION

The format this build writes.

LOG_PAGE_LEN

A log page is 32 MiB laid across contiguous segments.

MAGIC

The sixteen bytes at offset zero of every .yo file.

MAX_PAGE_SIZE

The largest legal segment size.

MIN_PAGE_SIZE

The smallest legal segment size.

MIN_READER_VERSION

The lowest reader version that can read what this build writes.

RECORD_ALIGN

Records are eight byte aligned, so every length rounds up to this.

SUPERBLOCK_LEN

A superblock slot, and therefore the offset of the second one.

Functions

align_up

n rounded up to the next multiple of RECORD_ALIGN.

checksum_skipping

CRC32C over bytes, with the four bytes at skip treated as zero.

get_u8

Reads a u8 at off, or 0 if the slice is too short.

get_u16

Reads a little endian u16 at off, or 0 if the slice is too short.

get_u32

Reads a little endian u32 at off, or 0 if the slice is too short.

get_u64

Reads a little endian u64 at off, or 0 if the slice is too short.

is_legal_page_size

Is n a legal segment size?

put_u8

Writes v at off. Does nothing if the slice is too short.

put_u16

Writes v little endian at off. Does nothing if the slice is too short.

put_u32

Writes v little endian at off. Does nothing if the slice is too short.

put_u64

Writes v little endian at off. Does nothing if the slice is too short.