pub struct LiveTable { /* private fields */ }Expand description
A table of rows identified by a key column, updated by Arrow record batches, with views (filtered, sorted) that follow every change.
Storage is log-structured: an upserted batch is kept as it is (its arrays shared, not
copied) and each key points at its latest row. Superseded rows are dropped by compaction,
automatically (see TableOptions) or with compact. Every key has a
stable internal id, so compaction never reorders a view.
Views and grouped aggregates (GroupSpec) are maintained incrementally: changing k
rows costs each view about k sorted insertions and removals (a few binary searches over
contiguous memory plus moving one block of at most 1,024 entries), and each group spec as
much per min or max column plus O(log groups) per row, not a re-sort or a recount; a
change touching more than an eighth of the rows rebuilds them instead. Reading a view
(View::row, O(log n)) never allocates; Groups::to_batch does.
Every column of the table is nullable, whatever the schema given says; batches must have the table’s column names and data types, in order.
Implementations§
Source§impl LiveTable
impl LiveTable
Sourcepub fn snapshot(&self) -> Result<Vec<RecordBatch>, DataError>
pub fn snapshot(&self) -> Result<Vec<RecordBatch>, DataError>
The live rows (the latest values of each key) as record batches, all with the table’s schema (stored batches keep the schema they arrived with, whose columns may not be nullable), to hand to something that reads Arrow: a SQL engine, a file writer, another thread (the table itself is used from one thread at a time). The columns of stored batches whose rows are all live are shared, not copied; the others are filtered down to their live rows. Rows come in no particular order.
A table without live rows gives no batches: take the schema from
schema. A table fed many small batches gives as many small ones;
compact first to have them merged (and superseded rows dropped).
The batches are a snapshot: later upserts and deletes do not change them. Taking one allocates: a bit per stored row of partly live batches, and their live rows’ copies.
§Errors
DataError::Arrow if Arrow cannot filter a batch or give it the table’s schema
(neither should happen to rows the table accepted).
Source§impl LiveTable
impl LiveTable
Sourcepub fn new(schema: SchemaRef, key: &str) -> Result<Self, DataError>
pub fn new(schema: SchemaRef, key: &str) -> Result<Self, DataError>
An empty table of schema, keyed by the column named key (integers or strings),
with default TableOptions.
§Errors
Sourcepub fn with_options(
schema: SchemaRef,
key: &str,
options: TableOptions,
) -> Result<Self, DataError>
pub fn with_options( schema: SchemaRef, key: &str, options: TableOptions, ) -> Result<Self, DataError>
An empty table of schema, keyed by the column named key, managed with options.
§Errors
Sourcepub fn instance(&self) -> u64
pub fn instance(&self) -> u64
Distinguishes this table object from every other one in the process, clones
included: something caching per table (a grid’s formatters) compares it with
generation.
Sourcepub fn batches(&self) -> &[RecordBatch]
pub fn batches(&self) -> &[RecordBatch]
Sourcepub fn generation(&self) -> u64
pub fn generation(&self) -> u64
Counts compactions: when it changes, batches were rewritten and earlier RowRefs
are invalid (views stay valid).
Sourcepub fn compaction_error(&self) -> Option<&DataError>
pub fn compaction_error(&self) -> Option<&DataError>
Why the last automatic compaction failed, if it did and no compaction has succeeded since. The table stays correct, only larger; automatic compaction tries again once twice as many rows are superseded.
Sourcepub fn record_of_key(&self, key: &Scalar) -> Option<RecordId>
pub fn record_of_key(&self, key: &Scalar) -> Option<RecordId>
The record of key key, if it is live.
Sourcepub fn row_of(&self, record: RecordId) -> Option<RowRef>
pub fn row_of(&self, record: RecordId) -> Option<RowRef>
Where the row of record is, or None if its key was deleted.
Sourcepub fn upsert(&mut self, batch: RecordBatch) -> Result<Changes, DataError>
pub fn upsert(&mut self, batch: RecordBatch) -> Result<Changes, DataError>
Inserts batch’s rows, replacing the rows of keys already present; within the batch
the last row for a key wins. Views follow.
§Errors
A column error if batch has other column names or data types,
DataError::NullKey if a row has no key, DataError::TooManyRows past 2^32 live
keys or rows in a batch. Nothing changes on error.
Sourcepub fn delete(&mut self, keys: &dyn Array) -> Result<Changes, DataError>
pub fn delete(&mut self, keys: &dyn Array) -> Result<Changes, DataError>
Deletes the rows of keys (an array of the key column’s kind; nulls and unknown keys
are skipped). Views follow.
§Errors
DataError::KeyType if keys is not of the key column’s kind: nothing changes.
Sourcepub fn compact(&mut self) -> Result<(), DataError>
pub fn compact(&mut self) -> Result<(), DataError>
Rewrites the stored batches with only the live rows, in batches of up to 65,536 rows
(fewer where one batch cannot hold them: more distinct strings than a dictionary
column’s key type indexes, over 2 GiB of strings). Dictionary columns of strings are
re-encoded with the strings their rows use. Views keep their order;
generation increases.
§Errors
DataError::Arrow if Arrow cannot build a batch of even 128 of the rows; nothing
changes.
Sourcepub fn add_view(&mut self, spec: ViewSpec) -> Result<ViewId, DataError>
pub fn add_view(&mut self, spec: ViewSpec) -> Result<ViewId, DataError>
Adds a view of the rows matching spec, filled from the current rows.
§Errors
DataError::UnknownColumn, DataError::UnsupportedColumn or
DataError::LiteralMismatch if the spec does not fit the schema.
Sourcepub fn remove_view(&mut self, id: ViewId) -> bool
pub fn remove_view(&mut self, id: ViewId) -> bool
Removes a view; false if it was already gone or belongs to another table. Its id
never names another view (its storage may be reused).
Sourcepub fn view(&self, id: ViewId) -> Option<View<'_>>
pub fn view(&self, id: ViewId) -> Option<View<'_>>
The rows of view id, in order; None if it was removed or belongs to another
table.
Sourcepub fn add_groups(&mut self, spec: GroupSpec) -> Result<GroupsId, DataError>
pub fn add_groups(&mut self, spec: GroupSpec) -> Result<GroupsId, DataError>
Adds grouped aggregates of the rows matching spec, filled from the current rows.
§Errors
DataError::UnknownColumn or DataError::UnsupportedColumn for a key or an
aggregated column, DataError::NotNumeric for a sum or mean of a column that is
not numeric, DataError::DuplicateName if two result columns share a name,
DataError::LiteralMismatch for a filter that does not fit.
Sourcepub fn remove_groups(&mut self, id: GroupsId) -> bool
pub fn remove_groups(&mut self, id: GroupsId) -> bool
Removes grouped aggregates; false if already gone or another table’s. Their id
never names other aggregates (their storage may be reused).