Crate rudb_native 

Rudb’s single-file columnar snapshot format.

A committed directory names independently readable column pages. The first version handles scalar columns and one table; the file header already has two generation slots so an unfinished replacement directory cannot hide the last complete one.

Parts and stripes

A part is one appended chunk, which is a thousand rows, and it is the unit a scan decodes and hands to the pipeline. A stripe is sixty four parts, and it is the unit the directory describes and the unit the file is laid out in: one page per column per stripe, holding that column’s sixty four part payloads end to end.

The two are separate because they are sized by different pressures. A part wants to be small because it is a vector and vectors live in cache. A stripe wants to be large because everything the directory holds is per stripe and the directory is one buffer that has to be read and decoded before a single row can be answered. A hundred million rows of the hundred and five column ClickBench table is ninety seven thousand parts, and a directory with a page entry and a pair of bounds per part per column is several hundred megabytes, which is what made that load fail before this split existed. Sixty four parts to a stripe divides that by sixty four.

Where the parts of a page start is not in the directory either, for the same reason. Each stripe writes one index page holding a length and a checksum per part per column, and a reader preads the sixty four entries belonging to the column it wants. A scan reads the whole column page once and slices it; a sparse row fetch reads the index entries and then only the part it needs.

Structs

ColumnLayout

Where one column’s bytes went, taken from the directory rather than by reading pages.

FrequencyOccurrences

Sparse row ordinals covered by a numeric frequency candidate set.

Layout

Where a whole file’s bytes went.

Opening

What Reader::open read before it returned.

Reader

Reads committed native column pages without holding the table in memory.

Reads

What a reader has read, while it was being opened and since.

Stripe

One independently readable stripe of a table.

Table

The committed table directory.

Writer

Appends pages and commits a new directory for one table.