Struct Writer 

pub struct Writer { /* private fields */ }

Appends pages and commits a new directory.

One writer covers a whole file rather than one table. Writer::next closes the table it is on and opens another over the same file, and Writer::finish commits every table it has closed in one generation. That is what makes a checkpoint atomic across tables: there is one slot write at the end of it and a reader sees every table at the generation before it or every table at the generation after it.

Implementations

impl Writer

pub fn preparer(&self) -> Preparer

Something that encodes stripes for this writer without holding it. See Preparer::prepare.

It carries the profile the writer has when it is asked for, so a writer that is going to be given one with Writer::with_profile should be given it first.

pub fn merge(&mut self, prepared: Prepared) -> Result<Merged>

Takes a prepared stripe into the table’s dictionaries and statistics and counts its rows in.

This is the step that has to see the stripes one at a time, and it is a hash a distinct value of each varchar column rather than two a row. Whatever Writer::append_at left behind is written first as its own stripe, the same rule Writer::append_stripe has.

Errors

If the stripe was prepared for a table of other columns, or the buffered stripe cannot be written.

pub fn write(&mut self, paged: Paged) -> Result<()>

Writes a stripe whose pages are built.

Errors

If the stripe was built for a table of another width or cannot be written.

pub fn append_prepared(&mut self, prepared: Prepared) -> Result<()>

All four steps one after the other, for a caller with nobody to share the writer with.

Errors

The same as Writer::merge, Merged::pages and Writer::write.

impl Writer

pub fn open( path: impl AsRef<Path>, name: impl Into<String>, fields: Vec<Field>, ) -> Result<Self>

Opens a committed file and starts a table in the generation after the one it holds.

The tables already in the file are carried forward by name and by directory pointer, and their pages are not read. Nothing in the file is overwritten: the new table’s pages and the new catalog go on the end, past the catalog the committed generation points at, and the one write that is not an append is the slot in the header that Writer::finish does last.

That slot is the other one. A file committed at generation 1 is named by the slot at 16 and generation 2 writes the one at 44, so until the last four bytes of the commit land the file still reads as the generation before it, and a slot torn across a write fails its checksum and the reader falls back to the one beside it. This is what the second slot has always been for.

Errors

If the file has no valid committed directory, is not this build’s format, repeats the name of a table already in it that holds rows, has a field with no scalar encoding, or cannot be written.

pub fn create( path: impl AsRef<Path>, name: impl Into<String>, fields: Vec<Field>, ) -> Result<Self>

Creates a new v10 file and its first table.

Errors

If the file exists, a field has no scalar encoding, or the path cannot be written.

pub fn empty(path: impl AsRef<Path>, views: &[ViewEntry]) -> Result<()>

Creates a new file that holds no table at all, committed and ready to open.

A database somebody dropped the last table out of is still a database, and until this there was no way to write one down. Every other way into this file goes through a table, because Writer::create takes the first one and Writer::finish commits the one it is on, so a catalog with nothing in it could be read and not written. The format already allowed it: the catalog is a count and that many entries, and a count of nought encodes and decodes the same way every other count does, which is why nothing here is a version change.

It hands back nothing rather than a writer, because a writer with no table is a writer with nothing to append to. A file that is going to hold a table is Writer::create, and one that is going to have a table added to it later is Writer::open, which reads what this wrote the same way it reads any other generation.

It takes the views anyway, because a database with no table can still have views in it. A view over range or over another view names no table, so dropping the last table out of a database does not have to leave the catalog with nothing worth writing down.

Errors

If the file exists or the path cannot be written.

pub fn next(self, name: impl Into<String>, fields: Vec<Field>) -> Result<Self>

Closes the table this writer is on and starts another one in the same file.

Nothing is published here. The closed table’s directory is written so that the bytes are on disk and its span is known, and the catalog that names it is only written by Writer::finish, so a crash between two tables leaves the previous generation intact.

Errors

If the name repeats a table already closed, a field has no scalar encoding, or the table being closed cannot be written.

pub fn with_views(self, views: Vec<ViewEntry>) -> Self

Sets the views the next commit writes down, replacing whatever was carried forward.

It replaces rather than adds because the caller has the whole catalog in front of it and the writer does not. A view that was dropped is a view that is not in the list any more, and there is no other way for the writer to hear about that, since nothing else it is told about mentions views at all.

A writer that is never told anything writes back the views it read at Writer::open, so a checkpoint that only had a table to append does not quietly drop them.

pub fn with_profile(self, profile: Arc<LoadProfile>) -> Self

Charges the stages this writer runs to profile.

For the table being written now. Writer::next starts the next table without one, because a second table’s stripes charged to the first table’s load would be a profile of neither.

pub fn declare(self, clustering: Clustering) -> Result<Self>

Records the order this table’s rows are meant to be stored in.

The declaration goes in the table directory and comes back out of Table::clustering. Nothing here sorts anything, and nothing here checks that the rows handed to Writer::append arrive in the order this claims. That is deliberate for now: the thing that was missing was a place to write the order down, and a loader that honours the declaration is the next piece rather than this one.

The declaration applies to the table the writer is currently on, so it is set after Writer::next rather than once for the file.

Errors

If the declaration names a column this table does not have.

pub fn append(&mut self, chunk: &Chunk) -> Result<()>

Writes one chunk as independently readable column pages.

Errors

If its width or types differ from the declared table, or a page exceeds its bound.

pub fn append_at(&mut self, order: (u64, u64), chunk: &Chunk) -> Result<()>

Writes one chunk and records its source position for directory ordering.

Pages may be encoded by parallel pipeline instances and reach the file in completion order. The stripe they land in is sorted by this key at commit, and Self::finish rejects a sequence whose parts do not come out in source order once the stripes are sorted, because a stripe groups whatever arrived together and cannot put a late part back where it belongs.

Errors

The same as Self::append.

pub fn append_stripe(&mut self, parts: Vec<((u64, u64), Chunk)>) -> Result<()>

Writes a run of chunks as one stripe of its own.

Self::append_at decides where a stripe ends by watching the orders go past, which works when one caller hands over every chunk in source order and does not when several do. A writer being fed by more than one pipeline instance sees the orders interleave, and a stripe that ends every time two of them cross is a stripe of one or two parts.

So the grouping moves to the caller. Whoever is buffering hands over a run it already knows is contiguous and in order, and gets a stripe holding exactly that run. The orders still have to come out in source order once the stripes are sorted, which Self::finish checks, so the runs from different callers may interleave with each other but may not overlap.

Errors

The same as Self::append, and if the run is longer than STRIPE_PARTS.

pub fn finish(self) -> Result<Table>

Commits every table this writer has written and syncs the file before publishing its header slot.

The table handed back is the one the writer was on, which is the last of them. Callers that wrote several already know the others, since they named them.

Errors

If directory encoding, writing, or syncing fails.

pub fn restate(path: impl AsRef<Path>, views: &[ViewEntry]) -> Result<()>

Commits a generation that changes the views and leaves every table exactly where it is.

There was no way to do this before views existed, because everything that could change the catalog also wrote a table, so the only way to say something new about a file was to go through a table. A view is the first thing that can change on its own. Without this, adding a view to a database with eight tables in it would rewrite all eight, since the append path needs a table to append and the fallback is the whole file.

It is the same commit as Writer::finish with nothing appended before it. The table entries are carried forward by directory pointer the way an append carries them, the new catalog goes on the end, and the slot write at the end is what publishes it.

Errors

If the file has no valid committed directory, is not this build’s format, or cannot be written.

Trait Implementations

impl Debug for Writer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.