Struct ColumnStore 

pub struct ColumnStore<P: Pager> { /* private fields */ }

Columnar storage engine for managing Arrow-based data.

ColumnStore provides the primary interface for persisting and retrieving columnar data using Apache Arrow RecordBatches. It manages:

• Column descriptors and metadata (chunk locations, row counts, min/max values)
• Data type caching for efficient schema queries
• Index management (presence indexes, value indexes)
• Integration with the Pager for persistent storage

Namespaces

Columns are identified by LogicalFieldId, which combines a namespace, table ID, and field ID. This prevents collisions between user data, row IDs, and MVCC metadata:

• UserData: Regular table columns
• RowIdShadow: Internal row ID tracking
• TxnCreatedBy: MVCC transaction creation timestamps
• TxnDeletedBy: MVCC transaction deletion timestamps

Thread Safety

ColumnStore is Send + Sync and can be safely shared across threads via Arc. Internal state (catalog, caches) uses RwLock for concurrent access.

Test Harness Integration

• SQLite sqllogictest: Every upstream case exercises the column store, providing a compatibility baseline but not full parity with SQLite yet.
• DuckDB suites: Early dialect-specific tests stress MVCC and typed casts, informing future work rather than proving comprehensive DuckDB coverage.
• Hardening mandate: Failures uncovered by the suites result in storage fixes, not filtered tests, to preserve confidence in OLAP scenarios built atop this crate.

Implementations

impl<P> ColumnStore<P>

where P: Pager<Blob = EntryHandle> + Send + Sync,

pub fn open(pager: Arc<P>) -> Result<Self>

Opens or creates a ColumnStore using the provided pager.

Loads the column catalog from the pager’s root catalog key, or initializes an empty catalog if none exists. The catalog maps LogicalFieldId to the physical keys of column descriptors.

Errors

Returns an error if the pager fails to load the catalog or if deserialization fails.

pub fn write_hints(&self) -> ColumnStoreWriteHints

Return heuristics that guide upstream writers when sizing batches.

pub fn register_index( &self, field_id: LogicalFieldId, kind: IndexKind, ) -> Result<()>

Creates and persists an index for a column.

Builds the specified index type for all existing data in the column and persists it atomically. The index will be maintained automatically on subsequent appends and updates.

Errors

Returns an error if the column doesn’t exist or if index creation fails.

pub fn has_field(&self, field_id: LogicalFieldId) -> bool

Checks if a logical field is registered in the catalog.

pub fn unregister_index( &self, field_id: LogicalFieldId, kind: IndexKind, ) -> Result<()>

Removes a persisted index from a column.

Atomically removes the index and frees associated storage. The column data itself is not affected.

Errors

Returns an error if the column or index doesn’t exist.

pub fn data_type(&self, field_id: LogicalFieldId) -> Result<DataType>

Returns the Arrow data type of a column.

Returns the data type from cache if available, otherwise loads it from the column descriptor and caches it for future queries.

Errors

Returns an error if the column doesn’t exist or if the descriptor is corrupted.

pub fn update_data_type( &self, field_id: LogicalFieldId, new_data_type: &DataType, ) -> Result<()>

Updates the data type of an existing column.

This updates the descriptor’s data type fingerprint and cache. Note that this does NOT migrate existing data - the caller must ensure data compatibility.

Errors

Returns an error if the column doesn’t exist or if the descriptor update fails.

pub fn ensure_column_registered( &self, field_id: LogicalFieldId, data_type: &DataType, ) -> Result<()>

Ensures that catalog entries and descriptors exist for a logical column.

Primarily used when creating empty tables so that subsequent operations (like CREATE INDEX) can resolve column metadata before any data has been appended.

pub fn filter_row_ids<T>( &self, field_id: LogicalFieldId, predicate: &Predicate<T::Value>, ) -> Result<Vec<u64>>

where T: FilterDispatch,

Find all row IDs where a column satisfies a predicate.

This evaluates the predicate against the column’s data and returns a vector of matching row IDs. Uses indexes and chunk metadata (min/max values) to skip irrelevant data when possible.

Errors

Returns an error if the column doesn’t exist or if chunk data is corrupted.

pub fn filter_matches<T, F>( &self, field_id: LogicalFieldId, predicate: F, ) -> Result<FilterResult>

where T: FilterPrimitive, F: FnMut(T::Native) -> bool,

Evaluate a predicate against a column and return match metadata.

This variant drives a primitive predicate over the column and returns a FilterResult describing contiguous match regions. Callers can use the result to build paginated scans or gather row identifiers.

Arguments

• field_id: Logical column to filter.
• predicate: Callable invoked on each value; should be cheap and free of side effects.

Errors

Returns an error if the column metadata cannot be loaded or if decoding a chunk fails.

pub fn list_persisted_indexes( &self, field_id: LogicalFieldId, ) -> Result<Vec<IndexKind>>

List all indexes registered for a column.

Returns the types of indexes (e.g., presence, value) that are currently persisted for the specified column.

Errors

Returns an error if the column doesn’t exist or if the descriptor is corrupted.

pub fn total_rows_for_field(&self, field_id: LogicalFieldId) -> Result<u64>

Get the total number of rows in a column.

Returns the persisted row count from the column’s descriptor. This value is updated by append and delete operations.

Errors

Returns an error if the column doesn’t exist or if the descriptor is corrupted.

pub fn total_rows_for_table(&self, table_id: TableId) -> Result<u64>

Get the total number of rows in a table.

This returns the maximum row count across all user-data columns in the table. If the table has no persisted columns, returns 0.

Errors

Returns an error if column descriptors cannot be loaded.

pub fn user_field_ids_for_table(&self, table_id: TableId) -> Vec<LogicalFieldId>

Get all user-data column IDs for a table.

This returns the LogicalFieldIds of all persisted user columns (namespace UserData) belonging to the specified table. MVCC and row ID columns are not included.

pub fn remove_column(&self, field_id: LogicalFieldId) -> Result<()>

Remove a column from the column store catalog.

This removes the column descriptor entry from the in-memory catalog and persists the updated catalog. The actual column data pages remain in the pager but become unreachable and will be garbage collected on compaction.

Arguments

• field_id - The logical field ID of the column to remove.

Errors

Returns an error if the column doesn’t exist or if catalog persistence fails.

pub fn has_row_id( &self, field_id: LogicalFieldId, row_id: RowId, ) -> Result<bool>

Check whether a specific row ID exists in a column.

This uses presence indexes and binary search when available for fast lookups. If no presence index exists, it scans chunks and uses min/max metadata to prune irrelevant data.

Errors

Returns an error if the column doesn’t exist or if chunk data is corrupted.

pub fn append(&self, batch: &RecordBatch) -> Result<()>

Append a RecordBatch to the store.

The batch must include a rowid column (type UInt64) that uniquely identifies each row. Each other column must have field_id metadata mapping it to a LogicalFieldId.

Last-Write-Wins Updates

If any row IDs in the batch already exist, they are updated in-place (overwritten) rather than creating duplicates. This happens in a separate transaction before appending new rows.

Row ID Ordering

The batch is automatically sorted by rowid if not already sorted. This ensures efficient metadata updates and naturally sorted shadow columns.

Table Separation

Each batch should contain columns from only one table. To append to multiple tables, call append separately for each table’s batch (may be concurrent).

Errors

Returns an error if:

• The batch is missing the rowid column
• Column metadata is missing or invalid
• Storage operations fail

pub fn delete_rows( &self, fields: &[LogicalFieldId], rows_to_delete: &[RowId], ) -> Result<()>

Delete row positions for one or more logical fields in a single atomic batch.

The same set of global row positions is applied to every field in fields. All staged metadata and chunk updates are committed in a single pager batch.

pub fn verify_integrity(&self) -> Result<()>

Verifies the integrity of the column store’s metadata.

This check is useful for tests and debugging. It verifies:

1. The catalog can be read.
2. All descriptor chains are walkable.
3. The row and chunk counts in the descriptors match the sum of the chunk metadata.

Returns Ok(()) if consistent, otherwise returns an Error.

pub fn get_layout_stats(&self) -> Result<Vec<ColumnLayoutStats>>

Gathers detailed statistics about the storage layout.

This method is designed for low-level analysis and debugging, allowing you to check for under- or over-utilization of descriptor pages.

impl<P> ColumnStore<P>

where P: Pager<Blob = EntryHandle>,

pub fn scan( &self, field_id: LogicalFieldId, opts: ScanOptions, visitor: &mut dyn PrimitiveFullVisitor, ) -> Result<()>

Unified scan entrypoint configured by ScanOptions. Requires V to implement both unsorted and sorted visitor traits; methods are no-ops by default.

impl<P> ColumnStore<P>

where P: Pager<Blob = EntryHandle> + Send + Sync,

pub fn gather_rows( &self, field_ids: &[LogicalFieldId], row_ids: &[u64], policy: GatherNullPolicy, ) -> Result<RecordBatch>

Gathers multiple columns using a configurable null-handling policy. When GatherNullPolicy::DropNulls is selected, rows where all projected columns are null or missing are removed from the resulting batch.

pub fn gather_rows_with_schema( &self, field_ids: &[LogicalFieldId], row_ids: &[u64], policy: GatherNullPolicy, expected_schema: Option<Arc<Schema>>, ) -> Result<RecordBatch>

Gather rows with an optional expected schema for empty result sets.

When expected_schema is provided and row_ids is empty, the returned RecordBatch will use that schema instead of synthesizing one with all nullable fields. This ensures that non-nullable columns (e.g., PRIMARY KEYs) are correctly represented.

pub fn prepare_gather_context( &self, field_ids: &[LogicalFieldId], ) -> Result<MultiGatherContext>

pub fn gather_rows_with_reusable_context( &self, ctx: &mut MultiGatherContext, row_ids: &[u64], policy: GatherNullPolicy, ) -> Result<RecordBatch>

Gathers rows while reusing chunk caches and scratch buffers stored in the context.

This path amortizes chunk fetch and decode costs across multiple calls by retaining Arrow arrays and scratch state inside the provided context.

Trait Implementations

impl<P> Clone for ColumnStore<P>

where P: Pager<Blob = EntryHandle> + Send + Sync,

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<P: Pager> ColumnStoreDebug for ColumnStore<P>

fn render_storage_as_formatted_string(&self) -> String

Renders the entire physical layout of the store into a formatted ASCII table string.

fn render_storage_as_dot( &self, batch_colors: &HashMap<PhysicalKey, usize>, ) -> String

Renders the physical layout of the store into a Graphviz .dot file string. The caller provides a map to color nodes based on an external category, like batch number.