Struct SegmentManager 

pub struct SegmentManager { /* private fields */ }

Segment manager for a collection

Implementations

impl SegmentManager

pub fn try_for_each_entity_unordered( &self, limit_bytes: usize, callback: impl FnMut(&UnifiedEntity) -> bool, ) -> Result<(), StoreError>

Visit the physical base and resident replacements without an ID-order workspace. This is for order-independent metadata reductions, not SQL ordering. The host must fence writers throughout this reduction so a materialized base row cannot be counted twice. Callbacks must not mutate or re-enter this collection.

pub fn try_paged_entity_cursor( &self, snapshot: Option<SnapshotContext>, ) -> Result<Option<PagedEntityCursor>, StoreError>

The host must fence publication while capturing the base and overlay. This retains O(dirty overlay) state and decodes at most one base row.

pub fn try_paged_physical_cursor( &self, ) -> Result<Option<PagedEntityCursor>, StoreError>

Raw physical records and their frozen metadata, including superseded versions. Hosts fence publication while capturing this cursor.

pub fn try_paged_key_entity_cursor( &self, key: &str, snapshot: Option<SnapshotContext>, ) -> Result<Option<PagedEntityCursor>, StoreError>

Freeze only this key’s payloads and metadata, retaining other dirty IDs as masks. Every decoded candidate is checked against the logical key.

pub fn try_paged_key_physical_cursor( &self, key: &str, ) -> Result<Option<PagedEntityCursor>, StoreError>

Like the visible keyed cursor, but includes superseded physical history.

pub fn has_backing(&self) -> bool

pub fn backing_root_page(&self) -> Option<u32>

Identity of the immutable base currently installed by the host.

pub fn backing_read_bytes_max(&self) -> Option<usize>

The host applies the same row allowance to model-specific expansion such as decrypting a compressed Vault value after native record decode.

pub fn install_backing( &self, source: Arc<dyn CollectionRead + Send + Sync>, root_page: u32, format_version: u32, read_bytes_max: usize, total_entities: u64, )

The host must fence writers and publish the new durable tree first. Captured resident readers retain their old sources through weak tracking.

pub fn backing_count_at_root(&self, root_page: u32) -> Option<u64>

pub fn backing_row_is_shadowed(&self, id: EntityId) -> bool

Whether a projected base posting has been replaced or physically removed. The host must fence publication/mutations while merging the projected base with resident rows. This never reads a base payload.

pub fn try_get( &self, id: EntityId, limit_bytes: usize, ) -> Result<Option<UnifiedEntity>, StoreError>

Preserve I/O and decode failures instead of treating them as absent rows.

pub fn try_project_key_entity<T>( &self, key: &str, snapshot: Option<&SnapshotContext>, versioned: bool, limit_bytes: usize, project: impl FnMut(&UnifiedEntity) -> T, ) -> Result<Option<T>, StoreError>

Fallible keyed projection. Resident mode keeps the segment key indexes. Prepared bases seek physical candidates through their paired key index; unindexed legacy sources retain a one-row streaming fallback. Hosts must fence publication, and callbacks must not mutate or re-enter this collection.

pub fn try_for_each_entity( &self, limit_bytes: usize, callback: impl FnMut(&UnifiedEntity) -> bool, ) -> Result<(), StoreError>

Visit the merged physical view in ID order without hydrating the base. Only overlay identities and one decoded base record are retained. Like resident visitors, callbacks must not mutate or re-enter this collection.

pub fn try_page_in_id_order( &self, offset: usize, limit: usize, limit_bytes: usize, ) -> Result<(usize, Vec<UnifiedEntity>), StoreError>

Page the same base/overlay physical view used by fallible query reads. Prefix records are skipped without decoding; only the requested page is cloned. The manager’s counter includes base rows and resident mutations.

pub fn try_for_each_key_entity( &self, key: &str, callback: impl FnMut(&UnifiedEntity) -> bool, ) -> Result<(), StoreError>

Physical key history. Indexed bases avoid unrelated payload reads; resident and unindexed sources still verify each logical key.

pub fn try_scan_for_each( &self, snapshot: Option<&SnapshotContext>, limit_bytes: usize, callback: impl FnMut(&UnifiedEntity) -> bool, ) -> Result<(), StoreError>

pub fn try_scan_for_each_zoned_with_stats( &self, snapshot: Option<&SnapshotContext>, zone_preds: &[(&str, ZoneColPred<'_>)], limit_bytes: usize, callback: impl FnMut(&UnifiedEntity) -> bool, ) -> Result<SegmentScanStats, StoreError>

The disk base has no segment zone summaries yet: scan all candidates and retain the caller’s residual predicate. Resident mode keeps pruning.

pub fn try_scan_project_zoned_with_stats<T, F>( &self, snapshot: Option<&SnapshotContext>, zone_preds: &[(&str, ZoneColPred<'_>)], read_bytes_max: usize, project: F, ) -> Result<(Vec<T>, SegmentScanStats), StoreError>

where T: Send, F: Fn(&UnifiedEntity) -> Option<T> + Sync,

Preserve the resident projection fast path; cold mode streams candidates. Hosts own admission for retained projections independently of row decoding.

pub fn try_get_many( &self, ids: &[EntityId], read_bytes_max: usize, ) -> Result<Vec<Option<UnifiedEntity>>, StoreError>

Point-read allowance applies to each row. Hosts separately admit the retained output vector; this method is not a global result quota.

pub fn try_for_each_id( &self, ids: &[EntityId], read_bytes_max: usize, visit: impl FnMut(usize, &UnifiedEntity) -> bool, ) -> Result<(), StoreError>

pub fn try_get_metadata( &self, id: EntityId, limit_bytes: usize, ) -> Result<Option<Metadata>, StoreError>

Metadata reads carry the same I/O failure contract as entity reads.

pub fn backing_overlay( &self, encode: impl FnMut(&UnifiedEntity, Option<Metadata>) -> Option<Vec<u8>>, ) -> Option<(Arc<dyn CollectionRead + Send + Sync>, Arc<FrozenOverlay>)>

pub fn current_backing_overlay( &self, ) -> Option<(Arc<dyn CollectionRead + Send + Sync>, Arc<FrozenOverlay>)>

impl SegmentManager

pub fn new(collection: impl Into<String>) -> Self

Create a new segment manager

pub fn with_config(collection: impl Into<String>, config: ManagerConfig) -> Self

Create with custom configuration

pub fn get_or_init_schema( &self, named: &HashMap<String, Value>, ) -> Arc<Vec<String>>

Get or create the shared column schema from first row’s named fields.

pub fn column_schema(&self) -> Option<Arc<Vec<String>>>

Get the column schema if it exists.

pub fn collection(&self) -> &str

Get collection name

pub fn config(&self) -> &ManagerConfig

Get configuration

pub fn stats(&self) -> ManagerStats

Get statistics. Entity counts cover active segments; memory also covers retained sources and unpublished consolidation work.

pub fn resident_bytes(&self) -> u64

Approximate resident segment memory, including tombstones, retired sources, and the in-flight merge and its bookkeeping. This feeds budget admission.

pub fn memory_bytes(&self) -> u64

Resident segment payload/directory bytes and consolidation bookkeeping, including retired sources. Unlike resident_bytes, excludes tombstones.

pub fn reclaimable_bytes(&self) -> u64

Bytes a pressure-triggered consolidation could plausibly return.

pub fn next_entity_id(&self) -> EntityId

Generate a new entity ID

pub fn identity_counters(&self) -> (u64, u64)

Durable allocator state may be ahead of every surviving record.

pub fn restore_identity_counters(&self, entity_id_next: u64, row_id_next: u64)

pub fn sequence_id_next(&self) -> u64

pub fn restore_sequence_id_next(&self, next: u64)

pub fn next_row_id(&self) -> u64

Generate a per-table sequential row ID (1, 2, 3… per collection)

pub fn reserve_row_ids(&self, n: u64) -> Range<u64>

Reserve n contiguous per-table row IDs with one atomic fetch_add. Caller assigns row_id = start + i per entity. Saves N-1 atomic RMWs on bulk inserts (25k atomics → 1).

pub fn register_row_id(&self, id: u64)

Advance the per-table row_id counter to at least id + 1. Called during load to restore the counter from existing data.

pub fn insert(&self, entity: UnifiedEntity) -> Result<EntityId, SegmentError>

Insert a new entity

pub fn insert_batch( &self, entities: Vec<UnifiedEntity>, ) -> Result<Vec<EntityId>, SegmentError>

Insert multiple entities (batch) — sequential, one lock per item.

pub fn bulk_insert( &self, entities: Vec<UnifiedEntity>, ) -> Result<Vec<EntityId>, SegmentError>

Turbo bulk insert — single lock acquisition for the entire batch. Skips bloom filter and cross-ref indexing for maximum speed.

pub fn get(&self, id: EntityId) -> Option<UnifiedEntity>

Get an entity by ID — scans growing then sealed segments.

pub fn get_many(&self, ids: &[EntityId]) -> Vec<Option<UnifiedEntity>>

Batch-fetch multiple entities by ID in a single lock acquisition per segment.

For indexed-scan result sets (up to ~5000 ids from range/bitmap lookup) this is 2-3 lock acquisitions total vs N×3 with individual get() calls.

pub fn for_each_id<F>(&self, ids: &[EntityId], f: F)

where F: FnMut(usize, &UnifiedEntity),

Visitor-pattern batch fetch. Invokes f(&UnifiedEntity, usize_index) for each id that resolves, never cloning the entity.

Used by scan hot paths (select_range, select_filtered) that materialize each entity into an output record and don’t need an owned UnifiedEntity. Eliminates ~20% of scan CPU spent in UnifiedEntity::clone when get_batch is followed by runtime_table_record_lean(entity).

The closure runs while the segment read lock is held, so it must be short — avoid doing I/O or taking unrelated locks in f.

pub fn update(&self, entity: UnifiedEntity) -> Result<(), SegmentError>

Update an entity

pub fn update_with_metadata( &self, entity: UnifiedEntity, metadata: Option<&Metadata>, ) -> Result<(), SegmentError>

Update an entity and, optionally, replace its metadata while holding the segment write lock only once.

pub fn update_hot( &self, entity: UnifiedEntity, modified_columns: &[String], ) -> Result<(), SegmentError>

HOT-update: like update but skips index work for unchanged columns. modified_columns is the list of column names actually changed by the UPDATE statement — lets us skip pk_index and cross_ref when safe.

pub fn update_hot_with_metadata( &self, entity: UnifiedEntity, modified_columns: &[String], metadata: Option<&Metadata>, ) -> Result<(), SegmentError>

HOT-update an entity and, optionally, replace its metadata while holding the segment write lock only once.

pub fn update_hot_batch_with_metadata<'a, I>( &self, items: I, ) -> Result<(), SegmentError>

where I: IntoIterator<Item = (&'a UnifiedEntity, &'a [String], Option<&'a Metadata>)>,

Batch HOT-update multiple entities while holding the growing-segment write lock only once when possible.

pub fn delete(&self, id: EntityId) -> Result<bool, SegmentError>

Delete an entity

pub fn delete_batch( &self, ids: &[EntityId], ) -> Result<Vec<EntityId>, SegmentError>

pub fn get_metadata(&self, id: EntityId) -> Option<Metadata>

Get metadata for an entity

pub fn set_metadata( &self, id: EntityId, metadata: Metadata, ) -> Result<(), SegmentError>

Set metadata for an entity

pub fn seal_current(&self) -> Result<SegmentId, SegmentError>

Seal the current growing segment

pub fn force_seal(&self) -> Result<Option<SegmentId>, SegmentError>

Force seal (for testing/manual control)

pub fn all_visible_fraction(&self) -> f64

Fraction of “pages” in sealed segments that are marked all-visible.

Sealed segments are immutable so all their rows are safe for index-only scans. The growing segment is never counted (writes may be in-flight). Uses rows_per_page = 256 (matching 8 KB pages with ~32-byte rows).

Returns a value in [0.0, 1.0]. 1.0 when all sealed rows are visible; 0.0 when there are no sealed segments.

pub fn for_each_entity<F>(&self, callback: F)

where F: FnMut(&UnifiedEntity) -> bool,

Iterate over all entities in-place without collecting into a Vec.

The callback receives a reference to each entity. Return true to continue iteration, false to stop early (e.g. when a LIMIT is reached). This avoids the allocation and cloning overhead of query_all.

pub fn fold_entities_parallel<T, FInit, FFold, FReduce>( &self, init: FInit, fold: FFold, reduce: FReduce, ) -> T

where T: Send, FInit: Fn() -> T + Send + Sync, FFold: Fn(T, &UnifiedEntity) -> T + Send + Sync, FReduce: Fn(T, T) -> T + Send + Sync,

Parallel fold across all entities. Each sealed segment is processed on its own rayon task; the growing segment stays on the caller thread (its read lock is briefly held).

• init builds a fresh accumulator per thread.
• fold mutates an accumulator with one entity at a time.
• reduce combines two accumulators into one.

The returned value is the reduction of every per-thread accumulator. Use this for aggregate-shape workloads (GROUP BY) where per-thread partial state can be merged cheaply.

NOTE: when there are 0 or 1 sealed segments, the parallel path is skipped and the work runs sequentially to avoid rayon overhead on tiny tables.

pub fn for_each_entity_zoned<F>( &self, zone_preds: &[(&str, ZoneColPred<'_>)], callback: F, )

where F: FnMut(&UnifiedEntity) -> bool,

Zone-map-aware iteration across all segments.

Like for_each_entity, but checks zone_preds against each segment’s column zone maps before iterating. Segments where any predicate can definitively prove no rows match are skipped entirely.

zone_preds: slice of (column_name, ZoneColPred) extracted from the WHERE clause. Empty slice → same behaviour as for_each_entity (no pruning).

pub fn for_each_entity_zoned_with_stats<F>( &self, zone_preds: &[(&str, ZoneColPred<'_>)], callback: F, ) -> SegmentScanStats

where F: FnMut(&UnifiedEntity) -> bool,

pub fn query_all_zoned<F>( &self, zone_preds: &[(&str, ZoneColPred<'_>)], filter: F, ) -> Vec<UnifiedEntity>

where F: Fn(&UnifiedEntity) -> bool + Sync,

Zone-map-aware parallel query.

Like query_all but applies zone_preds on the main thread to prune sealed segments before spawning workers — segments that provably contain no matching rows are skipped entirely.

Zone check runs single-threaded (it reads per-segment metadata, not row data), so it’s cheap. Surviving segments are then scanned in parallel using std::thread::scope when there are > 1 of them.

pub fn query_all_zoned_with_stats<F>( &self, zone_preds: &[(&str, ZoneColPred<'_>)], filter: F, ) -> (Vec<UnifiedEntity>, SegmentScanStats)

where F: Fn(&UnifiedEntity) -> bool + Sync,

pub fn scan_entity_visible( snapshot: Option<&SnapshotContext>, entity: &UnifiedEntity, ) -> bool

Visibility predicate for the scan_* family. Some delegates to the canonical entity_visible_with_context; None supports low-level callers that intentionally scan without a statement snapshot and keeps the documented resolver fallback — moderation gate plus hiding superseded physical versions — instead of showing every version.

pub fn scan<F>( &self, snapshot: Option<&SnapshotContext>, filter: F, ) -> Vec<UnifiedEntity>

where F: Fn(&UnifiedEntity) -> bool + Sync,

Scan every segment under an explicit MVCC snapshot.

Visibility is evaluated while each segment iterator is active, before the caller’s filter. Unlike the raw iteration methods, this entry point does not consult thread-local snapshot state, so it preserves the same view when invoked from a different thread.

pub fn scan_for_each<F>(&self, snapshot: Option<&SnapshotContext>, callback: F)

where F: FnMut(&UnifiedEntity) -> bool,

Visit every entity visible under an explicit MVCC snapshot.

Returning false from callback stops the scan early. Hidden entities never reach the callback and therefore cannot stop the scan.

pub fn scan_fold_parallel<T, FInit, FFold, FReduce>( &self, snapshot: Option<&SnapshotContext>, init: FInit, fold: FFold, reduce: FReduce, ) -> T

where T: Send, FInit: Fn() -> T + Send + Sync, FFold: Fn(T, &UnifiedEntity) -> T + Send + Sync, FReduce: Fn(T, T) -> T + Send + Sync,

Fold entities visible under an explicit MVCC snapshot in parallel.

pub fn scan_for_each_zoned_with_stats<F>( &self, snapshot: Option<&SnapshotContext>, zone_preds: &[(&str, ZoneColPred<'_>)], callback: F, ) -> SegmentScanStats

where F: FnMut(&UnifiedEntity) -> bool,

Visit zone-map candidates visible under an explicit MVCC snapshot.

pub fn scan_zoned_with_stats<F>( &self, snapshot: Option<&SnapshotContext>, zone_preds: &[(&str, ZoneColPred<'_>)], filter: F, ) -> (Vec<UnifiedEntity>, SegmentScanStats)

where F: Fn(&UnifiedEntity) -> bool + Sync,

Collect zone-map candidates visible under an explicit MVCC snapshot.

pub fn scan_project_zoned_with_stats<T, F>( &self, snapshot: Option<&SnapshotContext>, zone_preds: &[(&str, ZoneColPred<'_>)], project: F, ) -> (Vec<T>, SegmentScanStats)

where T: Send, F: Fn(&UnifiedEntity) -> Option<T> + Sync,

Project visible zone-map candidates while borrowing each source entity. Only the owned projection leaves the segment read guard. The callback must not mutate or re-enter the source collection.

pub fn query_all<F>(&self, filter: F) -> Vec<UnifiedEntity>

where F: Fn(&UnifiedEntity) -> bool + Sync,

Query across all segments. Uses parallel scanning for sealed segments when more than one sealed segment exists.

pub fn bloom_may_contain_key(&self, key: &[u8]) -> bool

Probe exact segment membership for a key without materializing matching entities.

Returns false only when every consulted segment proves absence. Missing segment state is conservative and returns true.

pub fn filter_metadata( &self, filters: &[(String, MetadataFilter)], ) -> Vec<EntityId>

Filter by metadata across all segments

pub fn get_by_kind(&self, kind: &str) -> Vec<UnifiedEntity>

Get entities by kind

pub fn count(&self) -> usize

Count entities

pub fn segment_ids(&self) -> Vec<SegmentId>

Get all segment IDs

pub fn drain_events(&self) -> Vec<LifecycleEvent>

Drain events. Kept for API compatibility; always returns empty because emit no longer buffers.

pub fn run_maintenance(&self) -> Result<(), SegmentError>

Run maintenance (would be called periodically in production)

pub fn pressure_consolidation_tick(&self) -> bool

Run one paced consolidation tick because admission pressure needs reclaimable bytes, even if the background thresholds have not fired.

Returns true when the call started or advanced a consolidation run.