Struct SSTableManager 

pub struct SSTableManager { /* private fields */ }

SSTable manager that handles multiple SSTable files

Implementations

impl SSTableManager

pub async fn refresh_tables(&self) -> Result<RefreshReport>

Re-scan the data directory and atomically apply added/removed SSTable generations to the held reader set. See the module docs for the full contract (snapshot-at-open, explicit refresh, in-flight isolation, fail-closed atomicity).

impl SSTableManager

pub async fn new( path: &Path, config: &Config, platform: Arc<Platform>, schema_registry: Option<Arc<RwLock<SchemaRegistry>>>, ) -> Result<Self>

Create a new SSTable manager

pub async fn new_from_discovered_paths( storage_path: &Path, table_dirs: Vec<PathBuf>, config: &Config, platform: Arc<Platform>, schema_registry: Option<Arc<RwLock<SchemaRegistry>>>, ) -> Result<Self>

Create a new SSTable manager from pre-discovered table directories

This method accepts a list of table directory paths (from DiscoveryService) and loads SSTables from those specific directories. It does not perform filesystem scanning beyond the provided directories - this avoids duplicate scanning when integrating with the discovery/engine lifecycle.

Use this method when you have pre-discovered table directories and want to avoid redundant filesystem scanning. Use new() when you want automatic discovery from a single base directory.

Arguments

• storage_path - Base storage path (used for context, not for scanning)
• table_dirs - List of table directory paths from DiscoveryService (e.g., /data/keyspace1/table1-abc123)
• config - Configuration
• platform - Platform abstraction

Returns

A new SSTableManager with SSTables loaded from the specified directories

Errors

Returns an error if any of the specified directories cannot be read. Individual SSTable loading errors are logged but do not fail the entire operation.

Example

use cqlite_core::storage::sstable::SSTableManager;
use cqlite_core::{Config, Platform};
use std::sync::Arc;
use std::path::PathBuf;

let config = Config::default();
let platform = Arc::new(Platform::new(&config).await?);

// Get table directories from DiscoveryService
let table_dirs = vec![
    PathBuf::from("/data/keyspace1/table1-abc123"),
    PathBuf::from("/data/keyspace1/table2-def456"),
];

let manager = SSTableManager::new_from_discovered_paths(
    &PathBuf::from("/data"),
    table_dirs,
    &config,
    platform,
    #[cfg(feature = "state_machine")]
    None,
).await?;

pub async fn create_from_memtable( &self, _data: Vec<(TableId, RowKey, ScanRow)>, ) -> Result<SSTableId>

pub async fn get( &self, table_id: &TableId, key: &RowKey, ) -> Result<Option<ScanRow>>

Get a value by key from all SSTables (simple version without tombstone merging)

Uses table_readers (keyed by fully-qualified "keyspace.table") so that only the SSTables for the requested table are searched (Issue #680). Same-named tables in different keyspaces (e.g. test_basic.simple_table and test_oa.simple_table) are now correctly distinguished.

Lookup order:

1. Exact match on the full table_id string (e.g. "test_basic.simple_table")
2. Unqualified table name (e.g. "simple_table") — for backward compatibility with flat/non-Cassandra directory layouts that have no keyspace parent.

pub async fn scan( &self, table_id: &TableId, start_key: Option<&RowKey>, end_key: Option<&RowKey>, limit: Option<usize>, schema: Option<&TableSchema>, ) -> Result<Vec<(RowKey, ScanRow)>>

Scan a range of keys from all SSTables for a table.

Arguments

• table_id - The table to scan
• start_key - Optional start key for range scan
• end_key - Optional end key for range scan
• limit - Optional limit on number of results
• schema - Optional table schema for schema-aware parsing. When provided, enables accurate type detection and avoids heuristic-based parsing. Strongly recommended for Cassandra 5.0+ formats.

Cross-generation reconciliation (last-write-wins + tombstone shadowing) is applied via the authoritative k-way merger when more than one SSTable generation backs the table and write-support + a schema are available; otherwise rows from each reader are concatenated. That concat fallback is the documented multi-generation limitation (Issue #883) and is now IDENTICAL across every feature build: the tombstones build takes exactly this path too (it no longer runs its own partition-keyed merge). So no build regresses relative to the default — a tombstones-without- write-support multi-generation read behaves the same as the default not(tombstones)-without-write-support build, and the prior tombstones “merge” it replaces was the row-collapsing bug, not real reconciliation.

Issue #1085: this is the SINGLE scan implementation for every feature build. The former #[cfg(feature = "tombstones")] variant grouped per-row results into a HashMap keyed on RowKey (which carries only the partition-key bytes, no clustering) and ran TombstoneMerger, so it collapsed all clustering rows of a partition into one — a full SELECT * over a clustered table returned ~one row per partition. Concatenating per-reader rows here (and reconciling only ACROSS generations) is correct for clustered tables in every build.

Lookup order (Issue #680):

1. Exact match on the full table_id string (e.g. "test_basic.simple_table")
2. Unqualified table name (e.g. "simple_table") — for backward compatibility with flat/non-Cassandra directory layouts that have no keyspace parent.

pub async fn scan_partition( &self, table_id: &TableId, partition_key: &[u8], schema: Option<&TableSchema>, ) -> Result<(Vec<(RowKey, ScanRow)>, bool)>

Partition-targeted scan: return only the rows for a single partition key, touching only the SSTables whose bloom filter / BTI trie admit the key.

This is the storage-layer fast path for a fully-constrained WHERE pk = ? (Issue #949). Rather than scanning every SSTable for the table and filtering in memory, it prunes the reader set with might_contain_partition — an O(1) bloom check for BIG format, an O(log n) trie walk for BTI — and only parses the surviving candidates. On a table backed by thousands of SSTables, a single-partition read drops from “open and scan all of them” to “scan only the handful that can hold the key”.

Output matches filtering the full scan result down to partition_key: the same per-reader parse and the same cross-generation reconciliation run, just over the pruned candidate set. Concretely, with more than one candidate generation this drives the authoritative k-way merge (write-support, schema present); the single-candidate and concat fallbacks behave exactly as the corresponding scan paths do — including sharing scan’s known multi-generation concat limitation (Issue #883) when the merge is unavailable. The caller still applies its own predicate evaluation, so any over-inclusion (e.g. a BTI prefix-collision candidate) is filtered out downstream.

Gated on not(tombstones) because the bloom/BTI prune fast path it relies on (scan_partition_clustering) is itself not(tombstones)-only. Under tombstones the executor falls back to a full scan + predicate filter (since #1085, scan is the same correct implementation in both builds, so the fallback is correct — just without the single-partition prune).

partition_key is the raw on-disk partition-key bytes produced by encode_partition_key_columns, which match the bytes the bloom filter, Index.db/BTI trie, and scan RowKeys are keyed on.

Within-SSTable seek (Issue #953): for the SINGLE-candidate case (the common point-lookup path) this seeks directly to the partition’s Data.db offset — resolved via the BTI Partitions.db trie or the BIG Index.db — and decodes ONLY that partition via scan_single_partition_clustering, instead of full-parsing the candidate and retaining one partition. The decode reuses the scan path’s parse_block_emit, so its output is byte-for-byte identical to scan(...).retain(matches_key); when the offset cannot be resolved authoritatively (no Index.db hit, or an unsupported format) it falls back to the full scan + retain for that candidate. The MULTI-candidate path is unchanged: it still reconciles via the k-way merge (or the per-candidate concat fallback), so cross-generation LWW / tombstone shadowing (#883) is preserved.

Returns (rows, engaged). On this build engaged is always true: the underlying scan_partition_clustering prunes the SSTable set via might_contain_partition before decoding, so a caller may honestly report a partition-targeted access path. The tombstones-build counterpart returns false because it has no prune (Epic #951, honest access paths).

pub async fn scan_partition_with_cell_metadata( &self, table_id: &TableId, partition_key: &[u8], schema: Option<&TableSchema>, ) -> Result<(Vec<(RowKey, ScanRow, HashMap<String, CellWriteMetadata>)>, bool)>

pub async fn scan_partition_clustering( &self, table_id: &TableId, partition_key: &[u8], clustering: Option<&ClusteringSlice>, schema: Option<&TableSchema>, ) -> Result<(Vec<(RowKey, ScanRow)>, bool)>

Clustering-slice-aware partition-targeted scan (Issue #954, Epic #951).

Identical to scan_partition but, when clustering is Some(slice) AND exactly one candidate SSTable admits the key AND that candidate’s single-partition seek can use its authoritative row index, the within-partition decode is bounded to the row-index block(s) covering the requested clustering range — so a WHERE pk = ? AND ck </>/= ? slice over a wide partition decodes O(matched rows + index block), not the whole partition.

Returns (rows, clustering_seek_engaged). clustering_seek_engaged is true only when the within-partition clustering narrowing actually bounded the decode (so the caller may report AccessPath::ClusteringSlice); it is false for the multi-candidate / merge / full-decode fallbacks, which still return correct rows for the honest PartitionLookup path. The rows are ALWAYS the full partition (or its clustering-narrowed superset): the caller’s post-scan evaluate_leaf applies the exact clustering bound, so output is byte-identical regardless of whether the seek engaged.

pub async fn scan_with_cell_metadata( &self, table_id: &TableId, start_key: Option<&RowKey>, end_key: Option<&RowKey>, limit: Option<usize>, schema: Option<&TableSchema>, ) -> Result<Vec<(RowKey, ScanRow, HashMap<String, CellWriteMetadata>)>>

Scan a table and return per-cell write metadata alongside row values.

Used when ProjectionFlags::include_cell_metadata is set (issue #693 — the WRITETIME/TTL threading bridge). Delegates to each reader’s scan_with_cell_metadata. When multiple readers serve the same table the results are concatenated; token-order sort and LIMIT are applied afterward.

Falls back to the regular scan with empty metadata when the reader does not surface metadata (non-V5CompressedLegacy paths).

Issue #1535: this is the single implementation for both the default / write-support build and the tombstones build. The former tombstones variant delegated to scan and returned empty metadata, so WRITETIME(col) / TTL(col) wrongly resolved to null under --features tombstones. The reader-level scan_with_cell_metadata surfaces the authoritative per-cell timestamp/TTL regardless of feature flags, so the fix is simply to use it in both builds. The cross-generation metadata merge stays gated on write-support (it needs the KWayMerger); without it the per-reader concatenation still surfaces each cell’s real metadata rather than fabricating or dropping it.

pub async fn partition_key_shape( &self, table_id: &TableId, ) -> Option<PartitionKeyShape>

Resolve the AUTHORITATIVE partition-key shape for table_id from the SSTable readers’ Statistics.db SerializationHeader (issue #1750).

This is the metadata a schema-less reader DOES have without a CQL schema: the SerializationHeader (surfaced on each reader’s header().columns) records, per column, the authoritative is_primary_key / is_clustering flags and the REAL names of the regular + static (non-key) columns. It does NOT record the partition-/clustering-key column NAMES — Cassandra never serialises those (they live in system_schema, absent schema-less), so the parser synthesises a placeholder pk name (build_partition_key_columns) that MUST NOT be trusted as an identity.

Returns PartitionKeyShape carrying ONLY authoritative facts: the count of partition-key components, the count of clustering keys, and the set of real non-key column names. The caller confirms a predicate column is the sole partition key BY ELIMINATION (single pk component, zero clustering keys, and the column absent from the non-key name set) — never by the synthesised name.

The shape is resolved across ALL readers serving the table, not just the first (issue #1750, roborev 3786). A multi-generation / schema-evolved table can add a REGULAR column in a LATER generation that is absent from an earlier generation’s header. Consulting only the first reader would MISS that column from the non-key name set, so a WHERE <later-gen-regular-col> = <val> could be MISCLASSIFIED as a partition-key seek. To prevent that:

• the non-key column names are UNIONed across every reader’s header, and
• the partition-key count, clustering-key count, and the single-component pk type+name must be CONSISTENT across all readers with a header.

Returns None when no reader for the table exposes a SerializationHeader OR when the readers’ key metadata is INCONSISTENT (a pk-count / clustering-count / single-pk type+name disagreement — which should never happen for one table but, if it did, means we cannot trust the shape). Both are fail-safe: the caller then keeps the honest full-scan path (all from SerializationHeaders, no heuristics — #28).

pub async fn scan_stream_materializes( &self, table_id: &TableId, _schema: Option<&TableSchema>, ) -> bool

See the tombstones variant above.

pub async fn scan_stream( &self, table_id: &TableId, start_key: Option<&RowKey>, end_key: Option<&RowKey>, schema: Option<&TableSchema>, buffer_size: usize, ) -> Result<Receiver<Result<(RowKey, ScanRow)>>>

Streaming scan (issue #790): merge per-SSTable streams lazily into a bounded output channel, in key (token) order, without materializing the whole result.

Each reader yields entries already in token order; a k-way merge over the per-reader heads produces globally ordered output while holding only one pending entry per SSTable. Live heap is bounded by buffer_size plus the number of SSTables, independent of total row count — the streaming analog of the materializing scan (concat + stable sort by key).

Multi-generation correctness (Issue #957)

The lazy per-reader k-way merge above is only the streaming analog of scan’s concat + sort path, which is correct for a single generation. When a table directory holds more than one SSTable generation, the same (partition, clustering) row can live in several generations and a row/cell tombstone in a newer generation suppresses only its own generation’s copy — so a pure key-ordered merge would emit overwritten rows twice and resurrect rows deleted in a later generation. scan avoids this by routing the multi-generation case through the authoritative KWayMerger (the same LWW + tombstone-shadowing k-way merge compaction uses); this streaming path must reconcile identically or execute() and execute_streaming() diverge.

Streaming the multi-generation merge (Issue #1579 / D3)

The multi-generation case runs [generation_merge::stream_generations_for_read], which drives that same KWayMerger on a blocking task and feeds each stepped partition’s live rows STRAIGHT into the bounded channel via blocking_send (backpressure preserved) — instead of collecting the ENTIRE reconciled table, sorting it, and only then dribbling it (the pre-D3 behaviour). Live heap is therefore O(one partition + channel), and time-to-first-row is O(first partition), not O(full merge). The merger already emits partitions in (token, key) order, byte-identical to scan’s sort_by_token_order, so the emitted order is unchanged (issue #1579 ordering guardrail). On a merger CONSTRUCTION error the driver reports back so this path FALLS BACK to the lazy per-reader streaming merge, mirroring scan’s fall-back-to-concat. The single-generation / no-schema / no-write-support cases keep the lazy streaming merge, which already matches scan’s concat path exactly and preserves LIMIT/backpressure.

pub async fn scan_stream_batched( &self, table_id: &TableId, start_key: Option<&RowKey>, end_key: Option<&RowKey>, schema: Option<&TableSchema>, buffer_size: usize, ) -> Result<Receiver<Result<Vec<(RowKey, ScanRow)>>>>

Batched streaming scan (issue #1592, Epic F/F2): the additive companion to scan_stream whose channel item is a Vec BATCH of entries. It forwards one batch per async wake instead of one row, undoing the per-row re-flattening on the public channel that the internal windowed pipeline (issue #1143) was built to avoid.

Content + order are identical to scan_stream: flattening the batches reproduces the per-row stream exactly.

• Single generation (one reader): the reader’s windowed batches are forwarded STRAIGHT THROUGH — no per-row channel is interposed, so the wake amortization survives end to end (the F2 win).
• Zero / multiple generations: reuses the fully-correct per-row scan_stream (cross-generation reconciliation + token-ordered k-way merge + empty case) and re-chunks its output into batches for the public channel. A generation-aware fully-batched merge is a deliberate follow-up (audit §Epic F); cross-generation correctness wins here.

pub async fn list_sstables(&self) -> Vec<SSTableId>

Get list of all SSTable IDs

pub async fn remove_sstable(&self, sstable_id: &SSTableId) -> Result<()>

Remove an SSTable

pub async fn stats(&self) -> Result<SSTableStats>

Get SSTable statistics

pub async fn set_schema_registry( &self, registry: Arc<RwLock<SchemaRegistry>>, ) -> Result<()>

Set the schema registry for schema-aware operations

This method stores the schema registry and applies it to all existing SSTable readers. Future readers loaded via load_existing_sstables or load_from_table_directories will also receive the schema registry during creation.

pub async fn merge_sstables( &self, _source_ids: Vec<SSTableId>, _target_id: SSTableId, ) -> Result<()>

Trait Implementations

impl Debug for SSTableManager

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.