Struct SegmentManager 

pub struct SegmentManager { /* private fields */ }

Segment manager for a collection

Implementations

impl SegmentManager

pub fn new(collection: impl Into<String>) -> Self

Create a new segment manager

pub fn with_config(collection: impl Into<String>, config: ManagerConfig) -> Self

Create with custom configuration

pub fn get_or_init_schema( &self, named: &HashMap<String, Value>, ) -> Arc<Vec<String>>

Get or create the shared column schema from first row’s named fields.

pub fn column_schema(&self) -> Option<Arc<Vec<String>>>

Get the column schema if it exists.

pub fn collection(&self) -> &str

Get collection name

pub fn config(&self) -> &ManagerConfig

Get configuration

pub fn stats(&self) -> ManagerStats

Get statistics. Entity counts cover active segments; memory also covers retained sources and unpublished consolidation work.

pub fn resident_bytes(&self) -> u64

Approximate resident segment memory, including tombstones, retired sources, and the in-flight merge and its bookkeeping. This feeds budget admission.

pub fn memory_bytes(&self) -> u64

Resident segment payload/directory bytes and consolidation bookkeeping, including retired sources. Unlike resident_bytes, excludes tombstones.

pub fn reclaimable_bytes(&self) -> u64

Bytes a pressure-triggered consolidation could plausibly return.

pub fn next_entity_id(&self) -> EntityId

Generate a new entity ID

pub fn next_row_id(&self) -> u64

Generate a per-table sequential row ID (1, 2, 3… per collection)

pub fn reserve_row_ids(&self, n: u64) -> Range<u64>

Reserve n contiguous per-table row IDs with one atomic fetch_add. Caller assigns row_id = start + i per entity. Saves N-1 atomic RMWs on bulk inserts (25k atomics → 1).

pub fn register_row_id(&self, id: u64)

Advance the per-table row_id counter to at least id + 1. Called during load to restore the counter from existing data.

pub fn insert(&self, entity: UnifiedEntity) -> Result<EntityId, SegmentError>

Insert a new entity

pub fn insert_batch( &self, entities: Vec<UnifiedEntity>, ) -> Result<Vec<EntityId>, SegmentError>

Insert multiple entities (batch) — sequential, one lock per item.

pub fn bulk_insert( &self, entities: Vec<UnifiedEntity>, ) -> Result<Vec<EntityId>, SegmentError>

Turbo bulk insert — single lock acquisition for the entire batch. Skips bloom filter and cross-ref indexing for maximum speed.

pub fn get(&self, id: EntityId) -> Option<UnifiedEntity>

Get an entity by ID — scans growing then sealed segments.

pub fn get_many(&self, ids: &[EntityId]) -> Vec<Option<UnifiedEntity>>

Batch-fetch multiple entities by ID in a single lock acquisition per segment.

For indexed-scan result sets (up to ~5000 ids from range/bitmap lookup) this is 2-3 lock acquisitions total vs N×3 with individual get() calls.

pub fn for_each_id<F>(&self, ids: &[EntityId], f: F)

where F: FnMut(usize, &UnifiedEntity),

Visitor-pattern batch fetch. Invokes f(&UnifiedEntity, usize_index) for each id that resolves, never cloning the entity.

Used by scan hot paths (select_range, select_filtered) that materialize each entity into an output record and don’t need an owned UnifiedEntity. Eliminates ~20% of scan CPU spent in UnifiedEntity::clone when get_batch is followed by runtime_table_record_lean(entity).

The closure runs while the segment read lock is held, so it must be short — avoid doing I/O or taking unrelated locks in f.

pub fn update(&self, entity: UnifiedEntity) -> Result<(), SegmentError>

Update an entity

pub fn update_with_metadata( &self, entity: UnifiedEntity, metadata: Option<&Metadata>, ) -> Result<(), SegmentError>

Update an entity and, optionally, replace its metadata while holding the segment write lock only once.

pub fn update_hot( &self, entity: UnifiedEntity, modified_columns: &[String], ) -> Result<(), SegmentError>

HOT-update: like update but skips index work for unchanged columns. modified_columns is the list of column names actually changed by the UPDATE statement — lets us skip pk_index and cross_ref when safe.

pub fn update_hot_with_metadata( &self, entity: UnifiedEntity, modified_columns: &[String], metadata: Option<&Metadata>, ) -> Result<(), SegmentError>

HOT-update an entity and, optionally, replace its metadata while holding the segment write lock only once.

pub fn update_hot_batch_with_metadata<'a, I>( &self, items: I, ) -> Result<(), SegmentError>

where I: IntoIterator<Item = (&'a UnifiedEntity, &'a [String], Option<&'a Metadata>)>,

Batch HOT-update multiple entities while holding the growing-segment write lock only once when possible.

pub fn delete(&self, id: EntityId) -> Result<bool, SegmentError>

Delete an entity

pub fn delete_batch( &self, ids: &[EntityId], ) -> Result<Vec<EntityId>, SegmentError>

pub fn get_metadata(&self, id: EntityId) -> Option<Metadata>

Get metadata for an entity

pub fn set_metadata( &self, id: EntityId, metadata: Metadata, ) -> Result<(), SegmentError>

Set metadata for an entity

pub fn seal_current(&self) -> Result<SegmentId, SegmentError>

Seal the current growing segment

pub fn force_seal(&self) -> Result<Option<SegmentId>, SegmentError>

Force seal (for testing/manual control)

pub fn all_visible_fraction(&self) -> f64

Fraction of “pages” in sealed segments that are marked all-visible.

Sealed segments are immutable so all their rows are safe for index-only scans. The growing segment is never counted (writes may be in-flight). Uses rows_per_page = 256 (matching 8 KB pages with ~32-byte rows).

Returns a value in [0.0, 1.0]. 1.0 when all sealed rows are visible; 0.0 when there are no sealed segments.

pub fn for_each_entity<F>(&self, callback: F)

where F: FnMut(&UnifiedEntity) -> bool,

Iterate over all entities in-place without collecting into a Vec.

The callback receives a reference to each entity. Return true to continue iteration, false to stop early (e.g. when a LIMIT is reached). This avoids the allocation and cloning overhead of query_all.

pub fn fold_entities_parallel<T, FInit, FFold, FReduce>( &self, init: FInit, fold: FFold, reduce: FReduce, ) -> T

where T: Send, FInit: Fn() -> T + Send + Sync, FFold: Fn(T, &UnifiedEntity) -> T + Send + Sync, FReduce: Fn(T, T) -> T + Send + Sync,

Parallel fold across all entities. Each sealed segment is processed on its own rayon task; the growing segment stays on the caller thread (its read lock is briefly held).

• init builds a fresh accumulator per thread.
• fold mutates an accumulator with one entity at a time.
• reduce combines two accumulators into one.

The returned value is the reduction of every per-thread accumulator. Use this for aggregate-shape workloads (GROUP BY) where per-thread partial state can be merged cheaply.

NOTE: when there are 0 or 1 sealed segments, the parallel path is skipped and the work runs sequentially to avoid rayon overhead on tiny tables.

pub fn for_each_entity_zoned<F>( &self, zone_preds: &[(&str, ZoneColPred<'_>)], callback: F, )

where F: FnMut(&UnifiedEntity) -> bool,

Zone-map-aware iteration across all segments.

Like for_each_entity, but checks zone_preds against each segment’s column zone maps before iterating. Segments where any predicate can definitively prove no rows match are skipped entirely.

zone_preds: slice of (column_name, ZoneColPred) extracted from the WHERE clause. Empty slice → same behaviour as for_each_entity (no pruning).

pub fn for_each_entity_zoned_with_stats<F>( &self, zone_preds: &[(&str, ZoneColPred<'_>)], callback: F, ) -> SegmentScanStats

where F: FnMut(&UnifiedEntity) -> bool,

pub fn query_all_zoned<F>( &self, zone_preds: &[(&str, ZoneColPred<'_>)], filter: F, ) -> Vec<UnifiedEntity>

where F: Fn(&UnifiedEntity) -> bool + Sync,

Zone-map-aware parallel query.

Like query_all but applies zone_preds on the main thread to prune sealed segments before spawning workers — segments that provably contain no matching rows are skipped entirely.

Zone check runs single-threaded (it reads per-segment metadata, not row data), so it’s cheap. Surviving segments are then scanned in parallel using std::thread::scope when there are > 1 of them.

pub fn query_all_zoned_with_stats<F>( &self, zone_preds: &[(&str, ZoneColPred<'_>)], filter: F, ) -> (Vec<UnifiedEntity>, SegmentScanStats)

where F: Fn(&UnifiedEntity) -> bool + Sync,

pub fn scan<F>( &self, snapshot: Option<&SnapshotContext>, filter: F, ) -> Vec<UnifiedEntity>

where F: Fn(&UnifiedEntity) -> bool + Sync,

Scan every segment under an explicit MVCC snapshot.

Visibility is evaluated while each segment iterator is active, before the caller’s filter. Unlike the raw iteration methods, this entry point does not consult thread-local snapshot state, so it preserves the same view when invoked from a different thread.

pub fn scan_for_each<F>(&self, snapshot: Option<&SnapshotContext>, callback: F)

where F: FnMut(&UnifiedEntity) -> bool,

Visit every entity visible under an explicit MVCC snapshot.

Returning false from callback stops the scan early. Hidden entities never reach the callback and therefore cannot stop the scan.

pub fn scan_fold_parallel<T, FInit, FFold, FReduce>( &self, snapshot: Option<&SnapshotContext>, init: FInit, fold: FFold, reduce: FReduce, ) -> T

where T: Send, FInit: Fn() -> T + Send + Sync, FFold: Fn(T, &UnifiedEntity) -> T + Send + Sync, FReduce: Fn(T, T) -> T + Send + Sync,

Fold entities visible under an explicit MVCC snapshot in parallel.

pub fn scan_for_each_zoned_with_stats<F>( &self, snapshot: Option<&SnapshotContext>, zone_preds: &[(&str, ZoneColPred<'_>)], callback: F, ) -> SegmentScanStats

where F: FnMut(&UnifiedEntity) -> bool,

Visit zone-map candidates visible under an explicit MVCC snapshot.

pub fn scan_zoned_with_stats<F>( &self, snapshot: Option<&SnapshotContext>, zone_preds: &[(&str, ZoneColPred<'_>)], filter: F, ) -> (Vec<UnifiedEntity>, SegmentScanStats)

where F: Fn(&UnifiedEntity) -> bool + Sync,

Collect zone-map candidates visible under an explicit MVCC snapshot.

pub fn query_all<F>(&self, filter: F) -> Vec<UnifiedEntity>

where F: Fn(&UnifiedEntity) -> bool + Sync,

Query across all segments. Uses parallel scanning for sealed segments when more than one sealed segment exists.

pub fn bloom_may_contain_key(&self, key: &[u8]) -> bool

Probe exact segment membership for a key without materializing matching entities.

Returns false only when every consulted segment proves absence. Missing segment state is conservative and returns true.

pub fn filter_metadata( &self, filters: &[(String, MetadataFilter)], ) -> Vec<EntityId>

Filter by metadata across all segments

pub fn get_by_kind(&self, kind: &str) -> Vec<UnifiedEntity>

Get entities by kind

pub fn count(&self) -> usize

Count entities

pub fn segment_ids(&self) -> Vec<SegmentId>

Get all segment IDs

pub fn drain_events(&self) -> Vec<LifecycleEvent>

Drain events. Kept for API compatibility; always returns empty because emit no longer buffers.

pub fn run_maintenance(&self) -> Result<(), SegmentError>

Run maintenance (would be called periodically in production)

pub fn pressure_consolidation_tick(&self) -> bool

Run one paced consolidation tick because admission pressure needs reclaimable bytes, even if the background thresholds have not fired.

Returns true when the call started or advanced a consolidation run.