Module record 

Record file format: extents of 512 byte blocks, payload codec, allocator. Record file: variable length memory records packed into 512 byte blocks.

Layout

block 0            file header (magic, version, block size, CRC)
block 1..end       extents, back to back

An extent is a run of contiguous blocks that starts with a 48 byte header:

[magic "MREC"][kind u8][pad 3][payload len u32][blocks u32]
[locator u64][memory id 16][payload crc u32][header crc u32]

A live extent holds one memory record (header plus encoded payload, padded to whole blocks); a free extent holds nothing but its header. Extents tile the file exactly: the extent at block b is followed by the extent at b + blocks. Every write keeps that chain intact (a new record carved from a free extent writes a free header for the remainder) and every write is WAL logged as the exact bytes it puts on disk, so after replay a scan can always walk the chain from block 1 without ever looking inside free space, where stale headers of dead records may linger.

A record’s size on disk is its encoded size rounded up to the next block, so a typical memory (1024 dimension embedding plus a few hundred bytes of text) occupies about 5 KB instead of a whole 64 KB page.

Locators

A record is addressed by its locator (PageId), a stable id that never changes for the life of the record. The engine keeps a locator to extent table in memory (persisted as the record map snapshot), so a record that outgrows its extent moves without its locator, and every cache or snapshot keyed by locator stays valid.

Structs

PageId

Stable locator of a memory record.

Constants

BLOCK_SIZE

Allocation unit of the record file. Records are rounded up to whole blocks, so the average waste is half a block per record.

DATA_FILE

File name of the record file inside the database directory.

EXTENT_HEADER_SIZE

Bytes of the header at the start of every extent.

WAL_FILE

File name of the record WAL inside the database directory.