Struct Writer 

pub struct Writer { /* private fields */ }

Appends pages and commits a new directory.

One writer covers a whole file rather than one table. Writer::next closes the table it is on and opens another over the same file, and Writer::finish commits every table it has closed in one generation. That is what makes a checkpoint atomic across tables: there is one slot write at the end of it and a reader sees every table at the generation before it or every table at the generation after it.

Implementations

impl Writer

pub fn open( path: impl AsRef<Path>, name: impl Into<String>, fields: Vec<Field>, ) -> Result<Self>

Opens a committed file and starts a table in the generation after the one it holds.

The tables already in the file are carried forward by name and by directory pointer, and their pages are not read. Nothing in the file is overwritten: the new table’s pages and the new catalog go on the end, past the catalog the committed generation points at, and the one write that is not an append is the slot in the header that Writer::finish does last.

That slot is the other one. A file committed at generation 1 is named by the slot at 16 and generation 2 writes the one at 44, so until the last four bytes of the commit land the file still reads as the generation before it, and a slot torn across a write fails its checksum and the reader falls back to the one beside it. This is what the second slot has always been for.

Errors

If the file has no valid committed directory, is not this build’s format, repeats the name of a table already in it, has a field with no scalar encoding, or cannot be written.

pub fn create( path: impl AsRef<Path>, name: impl Into<String>, fields: Vec<Field>, ) -> Result<Self>

Creates a new v10 file and its first table.

Errors

If the file exists, a field has no scalar encoding, or the path cannot be written.

pub fn next(self, name: impl Into<String>, fields: Vec<Field>) -> Result<Self>

Closes the table this writer is on and starts another one in the same file.

Nothing is published here. The closed table’s directory is written so that the bytes are on disk and its span is known, and the catalog that names it is only written by Writer::finish, so a crash between two tables leaves the previous generation intact.

Errors

If the name repeats a table already closed, a field has no scalar encoding, or the table being closed cannot be written.

pub fn declare(self, clustering: Clustering) -> Result<Self>

Records the order this table’s rows are meant to be stored in.

The declaration goes in the table directory and comes back out of Table::clustering. Nothing here sorts anything, and nothing here checks that the rows handed to Writer::append arrive in the order this claims. That is deliberate for now: the thing that was missing was a place to write the order down, and a loader that honours the declaration is the next piece rather than this one.

The declaration applies to the table the writer is currently on, so it is set after Writer::next rather than once for the file.

Errors

If the declaration names a column this table does not have.

pub fn append(&mut self, chunk: &Chunk) -> Result<()>

Writes one chunk as independently readable column pages.

Errors

If its width or types differ from the declared table, or a page exceeds its bound.

pub fn append_at(&mut self, order: (u64, u64), chunk: &Chunk) -> Result<()>

Writes one chunk and records its source position for directory ordering.

Pages may be encoded by parallel pipeline instances and reach the file in completion order. The stripe they land in is sorted by this key at commit, and Self::finish rejects a sequence whose parts do not come out in source order once the stripes are sorted, because a stripe groups whatever arrived together and cannot put a late part back where it belongs.

Errors

The same as Self::append.

pub fn append_stripe(&mut self, parts: Vec<((u64, u64), Chunk)>) -> Result<()>

Writes a run of chunks as one stripe of its own.

Self::append_at decides where a stripe ends by watching the orders go past, which works when one caller hands over every chunk in source order and does not when several do. A writer being fed by more than one pipeline instance sees the orders interleave, and a stripe that ends every time two of them cross is a stripe of one or two parts.

So the grouping moves to the caller. Whoever is buffering hands over a run it already knows is contiguous and in order, and gets a stripe holding exactly that run. The orders still have to come out in source order once the stripes are sorted, which Self::finish checks, so the runs from different callers may interleave with each other but may not overlap.

Errors

The same as Self::append, and if the run is longer than STRIPE_PARTS.

pub fn finish(self) -> Result<Table>

Commits every table this writer has written and syncs the file before publishing its header slot.

The table handed back is the one the writer was on, which is the last of them. Callers that wrote several already know the others, since they named them.

Errors

If directory encoding, writing, or syncing fails.

Trait Implementations

impl Debug for Writer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.