Crate densa_data 

Live Arrow tables: rows identified by a key, updated by record batches, and views that stay sorted and filtered as the rows change.

LiveTable takes Arrow RecordBatches from a feed, a query engine or a file, and upserts them by a key column; ViewSpec describes a view (a Filter and sort columns), which the table keeps up to date incrementally instead of recomputing it; GroupSpec does the same for grouped aggregates (count, sum, min, max, mean), grouped by key columns or by buckets of one (counts per minute, GroupSpec::group_by_time). A view’s rows are RowRefs into LiveTable::batches, so a display layer (such as densa-arrow’s grid) reads values straight from the Arrow arrays. Each key’s row is a RecordId while the key lives: a selection can follow it through updates, compactions and re-sorted views (View::position). LiveTable::snapshot hands the live rows to anything else that reads Arrow, such as a SQL engine. A Window keeps a table to its recent rows, by a time column. A LookupJoin gives each row of one table the columns of the row of another that it names (a trade its instrument), kept up to date as either changes.

This crate has no user-interface dependency.

use std::sync::Arc;
use arrow_array::{ArrayRef, Float64Array, RecordBatch, StringArray};
use densa_data::{CompareOp, Filter, LiveTable, Order, Scalar, ViewSpec};

let quotes = |symbols: Vec<&str>, prices: Vec<f64>| {
    RecordBatch::try_from_iter([
        ("symbol", Arc::new(StringArray::from(symbols)) as ArrayRef),
        ("price", Arc::new(Float64Array::from(prices)) as ArrayRef),
    ])
};
let mut table = LiveTable::new(quotes(vec![], vec![])?.schema(), "symbol")?;
let expensive = table.add_view(
    ViewSpec::new()
        .filter(Filter::compare("price", CompareOp::Gt, Scalar::Float(100.0)))
        .sort_by("price", Order::Descending),
)?;

table.upsert(quotes(vec!["AAA", "BBB", "CCC"], vec![150.0, 90.0, 120.0])?)?;
table.upsert(quotes(vec!["BBB"], vec![200.0])?)?; // BBB rises: it enters the view

let view = table.view(expensive).unwrap();
assert_eq!(view.len(), 3); // BBB 200, AAA 150, CCC 120

Re-exports

pub use arrow_array;

pub use arrow_schema;

Structs

Aggregate

One aggregate column of a GroupSpec: what it computes, over which column, and its name in the result.

Changes

What an upsert or delete did.

GroupSpec

Groups of rows with equal key values, and aggregates over each group.

Groups

The groups of a GroupSpec, ordered by key. Cheap to get.

GroupsId

Identifies grouped aggregates of one LiveTable (and of clones made after they were added, as for ViewId).

LiveTable

A table of rows identified by a key column, updated by Arrow record batches, with views (filtered, sorted) that follow every change.

LookupJoin

Joins the rows of a live table to the rows of another, the right table, that they name: each output row is a left row followed by the columns of the right row whose key is in the left row’s on column (nulls when there is none: a left outer join). The join keeps the right table; the output is an ordinary LiveTable the application owns (made by table), so views, groups, windows and grids work on it as on any table.

RecordId

A record: one key’s row for as long as the key is live, through updates, compactions and view changes. Deleting the key ends the record; inserting the key again starts a new one. Get one with LiveTable::record_of_key or View::record, find its row with LiveTable::row_of and its place in a view with View::position.

RowRef

Where a row’s values are: table.batches()[batch], row row. Valid until the next compaction, which LiveTable::upsert and LiveTable::delete may run: compare LiveTable::generation before reusing one.

TableOptions

How a LiveTable manages its storage.

View

The rows of a view, in its order. Cheap to get; reading never allocates.

ViewId

Identifies a view of one LiveTable. A clone of the table copies its views, so the id of a view added before cloning works on both; one added afterwards, only on its table.

ViewSpec

What a view shows: an optional filter and sort columns.

Window

Keeps a LiveTable to its recent rows: those whose value in a time column is within a span of the newest. Set it with LiveTable::set_window.

Enums

CompareOp

How a column compares with a literal.

DataError

Why a table or view operation was refused. Nothing changes when one is returned.

Filter

Which rows a view shows. Dates, times, timestamps and durations compare as their integer count of units (milliseconds since the epoch for Timestamp(Millisecond)).

Order

Sort direction.

Scalar

A literal value a column is compared with.