Struct SegmentManager 

pub struct SegmentManager<D: DirectoryWriter + 'static> { /* private fields */ }

Segment manager — coordinates segment commit, background merging, and trained structures.

SOLE owner of metadata.json. All metadata mutations go through state Mutex.

Implementations

impl<D: DirectoryWriter + 'static> SegmentManager<D>

pub fn new( directory: Arc<D>, schema: Arc<Schema>, metadata: IndexMetadata, merge_policy: Box<dyn MergePolicy>, term_cache_blocks: usize, max_concurrent_merges: usize, global_merge_permits: Arc<Semaphore>, merge_bp_time_budget: Option<Duration>, bp_memory_budget_bytes: usize, reorder_permits: Arc<Semaphore>, background_reorder_pool: Option<Arc<ThreadPool>>, ) -> Self

Create a new segment manager with existing metadata

pub fn background_cpu_pool(&self) -> Arc<ThreadPool>

Bounded rayon pool for background CPU (merge-time BP, manual reorder). Query scoring uses the global rayon pool; keeping background BP off it prevents a large merge from queueing every search behind gain passes.

pub fn begin_shutdown(&self)

Stop new indexing/merge/reorder operations from claiming segment IDs. Used as the first half of index deletion; the writer then joins its workers before Self::wait_for_shutdown drains remaining ownership.

pub async fn get_segment_ids(&self) -> Vec<String>

Get the current segment IDs

pub fn trained(&self) -> Option<Arc<TrainedVectorStructures>>

Get trained vector structures (lock-free via ArcSwap)

pub async fn load_and_publish_trained(&self)

Compatibility entry point: load the complete trained set or clear the published generation and log the validation failure.

pub async fn acquire_snapshot(&self) -> SegmentSnapshot

Acquire a snapshot of current segments for reading. The snapshot holds references — segments won’t be deleted while snapshot exists.

pub fn tracker(&self) -> Arc<SegmentTracker>

Get the segment tracker

pub fn directory(&self) -> Arc<D>

Get the directory

impl<D: DirectoryWriter + 'static> SegmentManager<D>

pub async fn commit( self: &Arc<Self>, new_segments: &[(String, u32)], ) -> Result<()>

Atomic commit: register new segments + persist metadata.

pub async fn maybe_merge(self: &Arc<Self>)

Evaluate merge policy and spawn background merges for all eligible candidates.

Atomicity: The entire filter → find_merges → spawn_merge sequence runs under the state lock to prevent a TOCTOU race where concurrent callers both see segments as eligible before either claims operation ownership. spawn_merge is non-blocking (just try_register + tokio::spawn), so holding the state lock through it is safe and sub-microsecond.

The hard merge semaphore is acquired before lifecycle ownership, so concurrent triggers cannot exceed configured merge capacity.

pub async fn abort_merges(&self)

Drain all in-flight merge tasks safely.

Tokio cannot abort a spawn_blocking closure once it has started. The old implementation aborted only the async wrapper and returned while merge-time BP still owned an OffsetWriter, allowing index deletion or orphan cleanup to race a live writer. Awaiting is the only sound generic behavior until every merge phase supports cooperative cancellation.

pub async fn wait_for_merging_thread(self: &Arc<Self>)

Wait for all current in-flight merges to complete.

pub async fn wait_for_all_merges(self: &Arc<Self>)

Wait for all eligible merges to complete, including cascading merges.

Drains current handles, then loops. Each completed merge auto-triggers maybe_merge (which pushes new handles) before its JoinHandle resolves, so by the time h.await returns all cascading handles are registered.

pub async fn wait_for_shutdown(self: &Arc<Self>)

Complete the second half of shutdown after the owning IndexWriter has been dropped. This drains tracked merges and then waits for every remaining guard, including optimizer reorders that are intentionally launched outside the writer lock.

pub async fn force_merge(self: &Arc<Self>) -> Result<()>

Force merge segments into the fewest possible segments, respecting max_segment_docs from the merge policy.

If the policy defines a max segment size, segments are merged in batches that stay within that limit. Otherwise, all segments are merged into one.

Each batch is registered in active_operations via an RAII guard to prevent maybe_merge from spawning a conflicting background merge.

pub async fn reorder_segments(self: &Arc<Self>) -> Result<()>

Reorder all segments via Recursive Graph Bisection (BP) for better BMP pruning.

Each segment is individually rebuilt with reordered BMP blocks. Non-BMP fields are copied unchanged via streaming file copy.

Uses active-operation ownership to prevent concurrent work on the same segment.

pub async fn unreordered_segment_ids(&self) -> Vec<String>

Get segment IDs that have not been reordered yet.

Excludes segments currently involved in a merge or reorder operation to avoid wasted work (the optimizer would skip them anyway).

pub async fn unreordered_segments(&self) -> Vec<(String, u32)>

Segments never reordered, with doc counts — for the optimizer to pick a size-appropriate BP budget.

pub async fn unconverged_segments(&self) -> Vec<(String, u32)>

Segments whose last BP pass hit its wall-clock budget before finishing (bp_converged == false). A warm-started follow-up pass deepens the ordering; the optimizer revisits these at low priority.

pub async fn unconverged_segments_below( &self, max_unconverged_passes: u32, ) -> Vec<(String, u32, u32)>

Unconverged segments still below a hard replacement-lineage work bound. Includes the persisted attempt count for scheduler diagnostics.

pub async fn reorder_single_segment( self: &Arc<Self>, seg_id: &str, rayon_pool: Option<Arc<ThreadPool>>, bp_budget: BpBudget, ) -> Result<bool>

Reorder a single segment via BP. Returns Ok(true) if reordered, Ok(false) if skipped.

Non-blocking: operation ownership prevents conflicts with background merges. Copies unchanged files and rebuilds only the sparse file with reordered BMP data.

pub async fn cleanup_orphan_segments(&self) -> Result<usize>

Clean up orphan segment files not registered in metadata.

Reads metadata, active-operation ownership, and snapshot-deferred deletions to determine which segments are legitimate. Filesystem deletion is asynchronous; in-flight outputs and retired sources still held by readers are both protected.