Crate llkv_column_map 

Columnar storage engine for LLKV.

This crate provides a low-level columnar storage layer that persists Apache Arrow RecordBatches to disk and supports efficient scans, filters, and updates. It serves as the foundation for llkv-table and higher-level query execution.

Architecture

The storage engine is organized into several key components:

• ColumnStore: Primary interface for storing and retrieving columnar data. Manages column descriptors, metadata catalogs, and coordinates with the pager for persistent storage.

• LogicalFieldId: Namespaced identifier for columns. Combines a namespace (user data, row ID shadow, MVCC metadata), table ID, and field ID into a single 64-bit value to prevent collisions.

• ScanBuilder: Builder pattern for constructing column scans with various options (filters, ordering, row ID inclusion).

• Visitor Pattern: Scans emit data through visitor callbacks rather than materializing entire columns in memory, enabling streaming and aggregation.

Storage Model

Data is stored in columnar chunks:

• Each column is identified by a LogicalFieldId
• Columns are broken into chunks for incremental writes
• Each chunk stores Arrow-serialized data plus metadata (row count, min/max values)
• Shadow columns track row IDs separately from user data
• MVCC columns (created_by, deleted_by) track transaction visibility

Namespaces

Columns are organized into namespaces to prevent ID collisions:

• UserData: Regular table columns
• RowIdShadow: Internal row ID tracking for each column
• TxnCreatedBy: MVCC transaction that created each row
• TxnDeletedBy: MVCC transaction that deleted each row

Thread Safety

ColumnStore is thread-safe (Send + Sync) with internal locking for catalog updates. Read operations can occur concurrently; writes are serialized through the catalog lock.

Macros and Type Dispatch

This crate provides macros for efficient type-specific operations without runtime dispatch overhead. See with_integer_arrow_type! for details.

Re-exports

pub use store::ColumnStore;

pub use store::IndexKind;

pub use store::ROW_ID_COLUMN_NAME;

pub use store::scan;

pub use store::scan::ScanBuilder;

Modules

debug

gather

Row gathering helpers for assembling Arrow arrays across chunks.

parallel

Helper utilities for Rayon thread-pool management.

store

ColumnStore facade and supporting modules.

types

Core type definitions for the columnar storage engine.

Macros

llkv_for_each_arrow_boolean

llkv_for_each_arrow_numeric

Invokes a macro for each supported Arrow numeric type.

with_integer_arrow_type

Dispatches to type-specific code based on an Arrow DataType.

Enums

Error

Unified error type for all LLKV operations.

Functions

ensure_supported_arrow_type

is_supported_arrow_type

supported_arrow_types

Type Aliases

Result

Result type alias used throughout LLKV.