differential_dataflow/columnar/mod.rs
1//! The columnar storage subsystem for differential dataflow.
2//!
3//! **Experimental.** API and internals are still settling. Expect breaking
4//! changes; do not rely on stability across releases.
5//!
6//! This is a self-contained storage variant: a columnar **data** core plus the
7//! two integration faces DD consumes, all derived from the same data. It borrows
8//! DD's abstractions (the [`Chunk`](crate::trace::chunk::Chunk) trait, the
9//! collection container traits) but could be sheared off without changing DD's
10//! own structure — a template for further storage variants.
11//!
12//! - **DATA** (this module's files) — the shared core:
13//! - [`layout`] — `ColumnarUpdate` / `ColumnarLayout` / `OrdContainer` / `Coltainer`
14//! (the columnar container as a DD `BatchContainer`).
15//! - [`updates`] — `UpdatesTyped<U>` trie, `Consolidating`, `UpdatesBuilder`, byte codec.
16//! - [`trie_merger`] — the trie-native survey/merge core the trace face delegates to.
17//! - [`collection`] — the COLLECTION face: `RecordedUpdates` + builder / Pact / operators.
18//! - [`trace`] — the TRACE face: `ColChunk` (a `Chunk` impl) + `Spine` / `Batcher` /
19//! `Builder` / `Chunker`.
20//!
21//! Known rough edge: `ContainerBytes` for `UpdatesTyped` is `unimplemented!()`.
22//! The wire-side container is `RecordedUpdates` (in [`collection`]), whose
23//! `ContainerBytes` is implemented; `UpdatesTyped` is the input-builder type and
24//! isn't shipped over channels.
25//!
26//! Files that touch both the local module path and the
27//! [`columnar`](https://docs.rs/columnar) crate should `use columnar as col;`
28//! to disambiguate.
29
30pub mod layout;
31pub mod updates;
32pub mod trie_merger;
33
34pub mod collection;
35pub mod trace;
36
37pub use updates::UpdatesTyped;
38
39/// Target size for update batches, in number of updates. The collection-side
40/// input builder and the trace-side `Chunker` aim to ship chunks within 1-2x this.
41pub const LINK_TARGET: usize = 64 * 1024;