Struct StorageEngine 

pub struct StorageEngine { /* private fields */ }

Main storage engine that coordinates all storage components

NOTE: Issue #176 removed write infrastructure (compaction, manifest). This is now a read-only storage layer focused on SSTable access.

Implementations

impl StorageEngine

pub async fn open( path: &Path, config: &Config, platform: Arc<Platform>, schema_registry: Option<Arc<RwLock<SchemaRegistry>>>, ) -> Result<Self>

Open a storage engine at the given path

This method discovers SSTables by scanning the storage directory. For pre-discovered SSTables, use open_with_sstables instead.

NOTE: Issue #176 removed write infrastructure (compaction, manifest). This is now a read-only storage layer focused on SSTable access.

pub async fn open_with_sstables( path: &Path, discovered_table_dirs: Vec<PathBuf>, config: &Config, platform: Arc<Platform>, schema_registry: Option<Arc<RwLock<SchemaRegistry>>>, ) -> Result<Self>

Open a storage engine with pre-discovered SSTable table directories

This method is used when SSTables have been discovered externally (e.g., by DiscoveryService) and allows the storage engine to be initialized with specific table directories rather than scanning the storage directory. Each table directory will be scanned for Data.db files.

Arguments

• path - Base storage path for manifest and SSTable operations
• discovered_table_dirs - Vector of table directory paths (each containing SSTable files)
• config - Storage configuration
• platform - Platform abstraction for I/O operations

Returns

A StorageEngine instance with all components initialized, including SSTable readers for all Data.db files found in the discovered table directories.

Example

let config = Config::default();
let platform = Arc::new(Platform::new(&config).await?);
let storage_path = Path::new("/var/lib/cqlite/storage");
let discovered_table_dirs = vec![
    PathBuf::from("/var/lib/cassandra/keyspace1/table1-abc123"),
    PathBuf::from("/var/lib/cassandra/keyspace1/table2-def456"),
];

let engine = StorageEngine::open_with_sstables(
    storage_path,
    discovered_table_dirs,
    &config,
    platform,
    #[cfg(feature = "state_machine")]
    None,
).await?;

pub async fn get( &self, table_id: &TableId, key: &RowKey, ) -> Result<Option<ScanRow>>

Get a value by key

pub async fn refresh(&self) -> Result<RefreshReport>

Re-scan the data directory and atomically apply added/removed SSTable generations to the held reader set (issue #1749).

Freshness contract

A StorageEngine (and the Database built on it) snapshots the discovered SSTable generations at open; it does not re-scan on its own. This is the ONLY way the reader set changes for a long-lived handle. Re-runs the same TOC/filename-based discovery open used — no content sniffing.

• Added generations become queryable; removed generations stop being queried; unchanged generations keep their warm parsed state.
• In-flight queries are unaffected: a scan already running holds its own Arc reader clones and completes against the pre-refresh set; queries started after the refresh see the new set.
• Atomic / fail-closed: if any newly discovered generation fails to open (e.g. a corrupt Statistics.db, issue #1626), the typed error is returned and the previously held reader set is left fully unchanged.

pub async fn scan( &self, table_id: &TableId, start_key: Option<&RowKey>, end_key: Option<&RowKey>, limit: Option<usize>, schema: Option<&TableSchema>, ) -> Result<Vec<(RowKey, ScanRow)>>

Scan a range of keys

Arguments

• table_id - The table to scan
• start_key - Optional start key for range scan
• end_key - Optional end key for range scan
• limit - Optional limit on number of results
• schema - Optional table schema for schema-aware parsing. When provided, enables accurate type detection and avoids heuristic-based parsing. Strongly recommended for Cassandra 5.0+ formats.

pub async fn scan_partition( &self, table_id: &TableId, partition_key: &[u8], schema: Option<&TableSchema>, ) -> Result<(Vec<(RowKey, ScanRow)>, bool)>

Partition-targeted scan for a fully-constrained WHERE pk = ? (Issue #949).

Returns only the rows for the single partition identified by the raw partition_key bytes, after pruning the SSTable set down to those whose bloom filter / BTI trie admit the key — so unrelated SSTables are never parsed. Output matches filtering the full scan result to the partition. Delegates to [SSTableManager::scan_partition] (which has a bloom-prune implementation for the default build and a scan-and-filter fallback for the tombstones build, so callers need no cfg branching).

Returns (rows, engaged): engaged is true only when the call actually pruned the SSTable set to partition candidates (the default build). The tombstones build returns false because it full-scans and retains with no prune, so the caller reports an honest fallback access path (Epic #951).

pub async fn partition_key_shape( &self, table_id: &TableId, ) -> Option<PartitionKeyShape>

Resolve the AUTHORITATIVE partition-key shape for table_id from the SSTable Statistics.db SerializationHeader (issue #1750).

Used by the SCHEMA-LESS point-read classifier to confirm — from authoritative metadata, never a synthesised pk name — that a col = <literal> predicate column really is the sole partition key before taking a by-key seek. Returns None when no reader exposes a SerializationHeader; delegates to [SSTableManager::partition_key_shape].

pub async fn scan_partition_clustering( &self, table_id: &TableId, partition_key: &[u8], clustering: Option<&ClusteringSlice>, schema: Option<&TableSchema>, ) -> Result<(Vec<(RowKey, ScanRow)>, bool)>

Clustering-slice-aware partition-targeted scan (Issue #954, Epic #951).

Like scan_partition but pushes a single-column clustering-key restriction (ck </>/= ? / two-bound range) down to a within-partition seek when the candidate’s authoritative row index supports it, so a wide-partition slice decodes O(matched rows + index) rather than the whole partition. Returns (rows, clustering_seek_engaged): the rows are the full partition (or a clustering-narrowed superset) so the caller’s post-scan filter yields byte-identical output, and the bool reports whether the clustering narrowing actually engaged (for the ClusteringSlice access path). Delegates to [SSTableManager::scan_partition_clustering].

pub async fn scan_partition_clustering_reverse( &self, table_id: &TableId, partition_key: &[u8], schema: Option<&TableSchema>, ) -> Result<Option<Vec<(RowKey, ScanRow)>>>

Reverse single-partition clustering scan for a BIG (nb) wide partition (Issue #1184). Returns Ok(Some(rows)) in DESCENDING clustering order when the BIG promoted-index reverse iterator applied, or Ok(None) to tell the caller to keep the in-memory ORDER BY DESC sort (small / BTI / multi-generation cases). Delegates to [SSTableManager::scan_partition_clustering_reverse].

pub async fn scan_partition_with_cell_metadata( &self, table_id: &TableId, partition_key: &[u8], schema: Option<&TableSchema>, ) -> Result<(Vec<(RowKey, ScanRow, HashMap<String, CellWriteMetadata>)>, bool)>

Partition-targeted, metadata-carrying scan for a fully-constrained WHERE pk = ? WRITETIME/TTL projection (Issue #962).

The metadata sibling of scan_partition: returns only the rows for the single partition identified by the raw partition_key bytes, WITH per-cell write metadata, after pruning the SSTable set down to the candidates whose bloom filter / BTI trie admit the key — so a SELECT WRITETIME(col) ... WHERE pk = ? never opens all N SSTables. Output matches filtering the full scan_with_cell_metadata result to the partition; cross-generation reconciliation runs over the pruned candidates. Delegates to [SSTableManager::scan_partition_with_cell_metadata].

Returns (rows, engaged): engaged is true only when the call pruned the SSTable set to partition candidates (the default build). The tombstones build returns false because it full-scans with metadata and retains with no prune, so the caller reports an honest fallback access path (Epic #951).

pub async fn scan_with_cell_metadata( &self, table_id: &TableId, start_key: Option<&RowKey>, end_key: Option<&RowKey>, limit: Option<usize>, schema: Option<&TableSchema>, ) -> Result<Vec<(RowKey, ScanRow, HashMap<String, CellWriteMetadata>)>>

Scan a table and return per-cell write metadata alongside row values.

Delegates to [SSTableManager::scan_with_cell_metadata]. Used when ProjectionFlags::include_cell_metadata is set (issue #693).

pub async fn scan_stream( &self, table_id: &TableId, start_key: Option<&RowKey>, end_key: Option<&RowKey>, schema: Option<&TableSchema>, buffer_size: usize, ) -> Result<Receiver<Result<(RowKey, ScanRow)>>>

Streaming scan (issue #790): return a bounded channel that yields (RowKey, ScanRow) entries lazily in key (token) order, instead of the materializing scan that returns the whole Vec.

Live heap is bounded by buffer_size rows rather than growing O(rows), so streaming a large SELECT * no longer holds the entire result set in memory at once. Delegates to SSTableManager::scan_stream.

pub async fn scan_stream_batched( &self, table_id: &TableId, start_key: Option<&RowKey>, end_key: Option<&RowKey>, schema: Option<&TableSchema>, buffer_size: usize, ) -> Result<Receiver<Result<Vec<(RowKey, ScanRow)>>>>

Batched streaming scan (issue #1592, Epic F/F2): additive companion to scan_stream that yields a Vec BATCH of (RowKey, ScanRow) entries per channel item instead of one entry, so a full-scan consumer is woken once per batch rather than once per row.

Content and order are identical to scan_stream — flattening the batches reproduces the per-row stream exactly. Backpressure is preserved (bounded channel). Delegates to SSTableManager::scan_stream_batched.

pub async fn scan_stream_materializes( &self, table_id: &TableId, schema: Option<&TableSchema>, ) -> bool

Reports whether scan_stream PRE-MATERIALIZES the full reconciled result for this table before returning the channel, rather than yielding rows lazily (issue #1577).

A bounded LIMIT consumer uses this to decide its QUERY_ROWS_SCANNED accounting: when it returns true the storage layer has already decoded the whole table (no decode-stop is possible and per-received-row counting would under-report), so the caller must charge the full decoded row count and take a materializing path. Delegates to SSTableManager::scan_stream_materializes.

pub async fn stats(&self) -> Result<StorageStats>

Get storage statistics

NOTE: Issue #176 removed compaction stats (compaction.rs deleted).

pub async fn shutdown(&self) -> Result<()>

Shutdown the storage engine

NOTE: Issue #176 removed compaction shutdown (compaction.rs deleted). Issue #175 removed flush operations (WAL/MemTable deleted).

pub async fn set_schema_registry( &self, registry: Arc<RwLock<SchemaRegistry>>, ) -> Result<()>

Set the schema registry for schema-aware operations

This method propagates the schema registry to the SSTable manager, which will apply it to all SSTable readers for schema-aware parsing.

Trait Implementations

impl Debug for StorageEngine

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.