Crate rudb_native 

Rudb’s single-file columnar snapshot format.

A committed directory names independently readable column pages. It has two levels: a catalog directory naming every table in the file, which is what a footer slot points at and what opening a database reads, and one directory per table under it holding that table’s stripes, pages and statistics. One slot write publishes all of them, so a commit is atomic across tables.

This version handles scalar columns; the file header has two generation slots so an unfinished replacement directory cannot hide the last complete one. See spec/storage-v3/12-many-tables-in-one-file.md.

Parts and stripes

A part is one appended chunk, which is a thousand rows, and it is the unit a scan decodes and hands to the pipeline. A stripe is sixty four parts, and it is the unit the directory describes and the unit the file is laid out in: one page per column per stripe, holding that column’s sixty four part payloads end to end.

The two are separate because they are sized by different pressures. A part wants to be small because it is a vector and vectors live in cache. A stripe wants to be large because everything the directory holds is per stripe and the directory is one buffer that has to be read and decoded before a single row can be answered. A hundred million rows of the hundred and five column ClickBench table is ninety seven thousand parts, and a directory with a page entry and a pair of bounds per part per column is several hundred megabytes, which is what made that load fail before this split existed. Sixty four parts to a stripe divides that by sixty four.

Where the parts of a page start is not in the directory either, for the same reason. Each stripe writes one index page holding a length and a checksum per part per column, and a reader preads the sixty four entries belonging to the column it wants. A scan reads the whole column page once and slices it; a sparse row fetch reads the index entries and then only the part it needs.

Structs

Catalog

Every table a native file holds, without the directory of any of them.

ColumnLayout

Where one column’s bytes went, taken from the directory rather than by reading pages.

Common

What a native table’s frequency synopsis says about one value, as the planner asks for it.

FrequencyOccurrences

Sparse row ordinals covered by a numeric frequency candidate set.

FrequencyPrefix

The values one column’s frequency synopsis lists, with a bound on everything it left out.

Layout

Where a whole file’s bytes went.

Opening

What Reader::open read before it returned.

Reader

Reads committed native column pages without holding the table in memory.

Reads

What a reader has read, while it was being opened and since.

Stripe

One independently readable stripe of a table.

Stripes

The bounds of a committed native table, as the planner asks for them.

Table

The committed table directory.

Writer

Appends pages and commits a new directory.

Constants

STRIPE_PARTS

Parts in one stripe.

Functions

distincts

How many distinct values each column of a native table holds, for the columns it can say.