Expand description
Rudb’s single-file columnar snapshot format.
A committed directory names independently readable column pages. The first version handles scalar columns and one table; the file header already has two generation slots so an unfinished replacement directory cannot hide the last complete one.
§Parts and stripes
A part is one appended chunk, which is a thousand rows, and it is the unit a scan decodes and hands to the pipeline. A stripe is sixty four parts, and it is the unit the directory describes and the unit the file is laid out in: one page per column per stripe, holding that column’s sixty four part payloads end to end.
The two are separate because they are sized by different pressures. A part wants to be small because it is a vector and vectors live in cache. A stripe wants to be large because everything the directory holds is per stripe and the directory is one buffer that has to be read and decoded before a single row can be answered. A hundred million rows of the hundred and five column ClickBench table is ninety seven thousand parts, and a directory with a page entry and a pair of bounds per part per column is several hundred megabytes, which is what made that load fail before this split existed. Sixty four parts to a stripe divides that by sixty four.
Where the parts of a page start is not in the directory either, for the same reason. Each stripe writes one index page holding a length and a checksum per part per column, and a reader preads the sixty four entries belonging to the column it wants. A scan reads the whole column page once and slices it; a sparse row fetch reads the index entries and then only the part it needs.
Structs§
- Frequency
Occurrences - Sparse row ordinals covered by a numeric frequency candidate set.
- Reader
- Reads committed native column pages without holding the table in memory.
- Stripe
- One independently readable stripe of a table.
- Table
- The committed table directory.
- Writer
- Appends pages and commits a new directory for one table.