Skip to main content

SegmentedStore

Struct SegmentedStore 

Source
pub struct SegmentedStore<S: Store> { /* private fields */ }
Expand description

A generic, durable, segmented mutable store.

Implementations§

Source§

impl<S: Store> SegmentedStore<S>

Source

pub fn open( dir: Arc<dyn Directory>, store: S, flush_threshold: usize, ) -> PersistenceResult<Self>

Open (or create) a store backed by dir, recovering any prior state from the checkpoint plus the write-ahead log.

flush_threshold is the buffered-add count that triggers a new segment.

Source

pub fn open_with_options( dir: Arc<dyn Directory>, store: S, opts: Options, ) -> PersistenceResult<Self>

Like Self::open, but with full Options (per-write durability, size-tiered compaction tuning, auto-compaction).

SyncPolicy::Fsync requires a filesystem-backed Directory; opening with it on a backend that lacks file_path() (e.g. an in-memory directory) returns an error rather than silently degrading to a flush.

Source

pub fn reader(&self) -> Reader<S>

A cloneable handle for concurrent snapshot reads while this writer mutates. Clone it across threads; each Reader::view is a consistent snapshot.

Source

pub fn add(&mut self, id: S::Id, item: S::Item) -> PersistenceResult<()>

Add (or re-add) an item. Durably logged before it becomes visible.

A re-add replaces the id’s previous value (last write wins), including a value deleted earlier. The buffered copy is replaced in place. A copy already sealed in a segment stays in that segment’s payload until a merge drops it, but is marked superseded: Self::is_live_in, View::is_live_in, and SegmentCatalog::is_live_in report it dead, so a consumer that filters each segment with them sees only the newest copy. Tracking sealed copies needs Store::item_ids (implemented by DefaultStore); without it only the buffered copy is replaced, and a sealed copy stays live next to the new one.

Source

pub fn extend( &mut self, items: impl IntoIterator<Item = (S::Id, S::Item)>, ) -> PersistenceResult<()>

Add (or re-add) many items, syncing the WAL once for the whole batch instead of per item. This is the bulk-ingest path (the build phase of an index, e.g. lexir/sporse loading a corpus): per-item flush is the dominant cost there, and one sync per batch is several times faster on a real disk.

The durability boundary is the batch’s final sync, so a crash mid-batch may leave a prefix of the batch durable (each item is an independently CRC-checked WAL record); recovery yields a consistent prefix, never a half-written item. Auto-compaction (if enabled) still runs when a segment seals during the batch.

Source

pub fn delete(&mut self, id: S::Id) -> PersistenceResult<()>

Tombstone an item. Durably logged before it takes effect.

Source

pub fn compact(&mut self) -> PersistenceResult<CompactionStats>

Merge ALL segments into one, dropping tombstoned ids and purging the tombstone set, then checkpoint. This is the full compaction; it is the only path that touches frozen (over-cap) segments and the only one that purges tombstones (a partial tier merge cannot, since a tombstoned id may live in a segment it did not merge).

Source

pub fn compact_tiers(&mut self) -> PersistenceResult<CompactionStats>

Run all currently-eligible size-tier merges (size-tiered compaction) per the TierConfig, persisting the result if anything merged. Segments are grouped into size buckets (a Cassandra-style bucket_low .. bucket_high band around a running average, with a Options::flush_threshold floor); a bucket with at least min_merge segments is merged (smallest first, up to max_merge at once, never exceeding max_merged_len items), dropping tombstoned ids from the merged output. Segments above max_merged_len / 2 are frozen out, so the largest segment is never rewritten here. The global tombstone set is NOT purged (only Self::compact can); a tombstoned id surviving in a frozen segment stays filtered by Self::is_live.

Scheduling is the consumer’s: call this when convenient (e.g. on a background thread) so merge latency stays off the ingest path, or set Options::auto_compact.

Source

pub fn force_merge_to( &mut self, max_segments: usize, ) -> PersistenceResult<CompactionStats>

Consolidate on demand: merge segments until at most max_segments remain, ignoring the size band, the per-bucket minimum, and the TierConfig cap (this is explicit user intent, e.g. before a read-heavy phase, so it may produce a segment larger than max_merged_len). Merging to a single segment (max_segments <= 1) also purges the tombstone set, exactly like Self::compact; merging to more keeps the set (an id may live in a segment the call did not merge). Smallest segments are merged first to keep the work down. A no-op when already at or below max_segments.

Source

pub fn has_eligible_tier(&self) -> bool

Whether at least one size bucket is eligible to merge right now.

Source

pub fn stored_len(&self) -> usize

Total items physically stored across segments and the buffer. This counts tombstoned-but-not-yet-purged items (segstore cannot see inside an opaque segment to subtract them); pair with Self::tombstone_count to gauge space amplification.

Source

pub fn epoch(&self) -> u64

The current WAL generation.

Source

pub fn space_amplification(&self) -> Option<f64>

Space amplification = stored items / live items, or None if any segment’s Store::live_len returns None (the default). A value near 1.0 means little dead data; a higher value means tombstoned/obsolete items are accumulating, reclaimable with Self::reclaim_tombstones or Self::compact.

Source

pub fn reclaim_tombstones( &mut self, min_live_ratio: f64, ) -> PersistenceResult<CompactionStats>

Merge segments whose live ratio (live_len / segment_len) is below min_live_ratio, reclaiming their dead (tombstoned) entries into one fresh segment. The cheap alternative to Self::compact when only a few segments are tombstone-heavy. Requires Store::live_len; a no-op if it returns None or nothing qualifies. Keeps the tombstone set (a reclaimed id may still live in a segment this did not touch); only compact purges it.

Source

pub fn segment_sizes(&self) -> Vec<usize>

Per-segment item counts, oldest segment first. The size distribution the consumer needs to watch the crossover signal (segment count + per-query fan-out vs corpus size) that says a multi-segment store is outgrowing the search-all-segments model.

Source

pub fn checkpoint(&mut self) -> PersistenceResult<()>

Snapshot the current segments + tombstones durably, then rotate the WAL: start a fresh epoch and delete the old log so it cannot grow unbounded.

The checkpoint is the commit point. It records the new epoch, so once it is durable, recovery replays only the new (initially empty) WAL; the old WAL is superseded. A crash after the checkpoint but before the old WAL is deleted is safe: recovery never reads the old epoch, and the stale file is GC’d on the next open.

If the manifest write fails, the new manifest may already be visible, so the store is poisoned: later writes and checkpoints return InvalidState. Reopen the store to recover.

Source

pub fn segments(&self) -> &[Arc<S::Segment>]

The immutable segments, oldest first (each derefs to S::Segment). Query these plus Self::buffer, filtering segment copies with Self::is_live_in. Segments are Arc-shared: an unchanged segment keeps its identity across mutations, so a consumer can cache per-segment state keyed by Arc::as_ptr and rebuild only new segments. For a consistent view while another thread mutates, use a Reader / View instead. For a cache key that also survives a restart, use Self::segment_ids.

Source

pub fn segment_ids(&self) -> &[u64]

The stable persistence id of each segment in Self::segments, same order. Unlike the segment’s Arc pointer (stable only within one process), this id names the segment’s segstore.seg.<id> file and is stable across checkpoints and restarts, so a consumer can key a persisted per-segment built-index cache on it and reload across a process restart instead of rebuilding from the raw payload. An id is never reused after its segment is compacted away.

Source

pub fn dir(&self) -> &Arc<dyn Directory> ⓘ

The directory this store is backed by, so a consumer can persist a built per-segment index alongside the segments. Write the index into the reserved sidecar name from Self::index_name, keyed by a stable Self::segment_ids id; segstore garbage-collects it when its segment is compacted away (on the same crash-safe schedule as the segment files). A missing or stale sidecar means “rebuild from the raw segment.” This is the persist-the-built-index hook: recovery loads the sidecar instead of rebuilding every consumer index.

Source

pub fn try_index_name(&self, id: u64, kind: &str) -> PersistenceResult<String>

The reserved sidecar file name for a per-segment index of the given kind (e.g. "hnsw"), keyed by a stable Self::segment_ids id. kind must be non-empty ASCII alphanumeric plus _ or -. It is a short namespace tag, not a compatibility proof; put full algorithm/config/input fingerprints in the sidecar bytes and rebuild from the raw segment when they mismatch. Use with Self::dir to persist/load a built index that segstore GCs in lockstep with the segment. Reader/searcher code that only has a View can use the module-level try_index_name helper with View::segment_ids.

Source

pub fn index_name(&self, id: u64, kind: &str) -> String

Infallible convenience wrapper around Self::try_index_name.

Panics if kind violates the documented sidecar-kind grammar.

Source

pub fn buffer(&self) -> &[(S::Id, S::Item)]

Live adds not yet flushed into a segment.

Source

pub fn is_live(&self, id: &S::Id) -> bool

Whether id is not tombstoned.

This is per-id: when a re-added id has copies in several segments, it is true for all of them. Filter a segment’s contents with Self::is_live_in instead. Every buffered item is live.

Source

pub fn is_live_in(&self, segment_id: u64, id: &S::Id) -> bool

Whether the copy of id in segment segment_id (an entry of Self::segment_ids) is live: id is not tombstoned and was not re-added after that segment sealed. At most one copy of an id passes this check (see Self::add).

Source

pub fn live_segment_of(&self, id: &S::Id) -> Option<u64>

The id of the segment holding id’s live sealed copy, or None if id is buffered, deleted, absent, or the store does not implement Store::item_ids.

A re-add or delete of id changes that segment’s live set, so a consumer that caches a per-segment index can call this first and drop just that segment’s cache entry, without scanning segments for the id.

Source

pub fn live_ids(&self, segment_id: u64) -> Option<LiveIds<S::Id>>
where S::Id: Ord,

Canonical live-id set of one sealed segment: the segment’s ids filtered by Self::is_live_in, sorted and deduped.

This is the fingerprint a sidecar cache keys on. Returns None if segment_id is not a current segment or the store’s Store implementation does not list ids (Store::item_ids). Buffered adds are not part of any segment’s live-id set.

Source

pub fn segment_count(&self) -> usize

Number of immutable segments.

Source

pub fn tombstone_count(&self) -> usize

Number of tombstoned ids.

Trait Implementations§

Source§

impl<S: Store> SidecarHost for SegmentedStore<S>

Source§

fn sidecar_dir(&self) -> &Arc<dyn Directory> ⓘ

The directory sidecars are written to.
Source§

fn try_sidecar_name( &self, segment_id: u64, kind: &str, ) -> PersistenceResult<String>

The reserved sidecar name for segment_id, validated against this host’s known segment set where the type can check it.
Source§

fn sidecar_name(&self, segment_id: u64, kind: &str) -> String

Infallible convenience wrapper around Self::try_sidecar_name. Read more
Source§

fn sidecar_file_path( &self, segment_id: u64, kind: &str, ) -> PersistenceResult<Option<PathBuf>>

Filesystem path for a sidecar, when the backend exposes paths.
Source§

fn sidecar_exists(&self, name: &str) -> bool

Whether a sidecar file exists.
Source§

fn read_sidecar(&self, name: &str) -> PersistenceResult<Option<Vec<u8>>>

Read a whole sidecar, or None if it is absent.
Source§

fn write_sidecar(&self, name: &str, bytes: &[u8]) -> PersistenceResult<()>

Atomically publish a sidecar; a torn write can never be observed.
Source§

fn delete_sidecar(&self, name: &str) -> PersistenceResult<()>

Delete a sidecar. Absent files are a no-op, like Directory::delete.

Auto Trait Implementations§

§

impl<S> !RefUnwindSafe for SegmentedStore<S>

§

impl<S> !Sync for SegmentedStore<S>

§

impl<S> !UnwindSafe for SegmentedStore<S>

§

impl<S> Freeze for SegmentedStore<S>
where S: Freeze, Vec<(<S as Store>::Id, <S as Store>::Item)>: Freeze, Vec<Arc<<S as Store>::Segment>>: Freeze, HashSet<<S as Store>::Id>: Freeze, HashMap<u64, HashSet<<S as Store>::Id>>: Freeze, Option<HashMap<<S as Store>::Id, u64>>: Freeze, Arc<RwLock<Arc<PubState<S>>>>: Freeze,

§

impl<S> Send for SegmentedStore<S>
where S: Send, Vec<(<S as Store>::Id, <S as Store>::Item)>: Send, Vec<Arc<<S as Store>::Segment>>: Send, HashSet<<S as Store>::Id>: Send, HashMap<u64, HashSet<<S as Store>::Id>>: Send, Option<HashMap<<S as Store>::Id, u64>>: Send, Arc<RwLock<Arc<PubState<S>>>>: Send,

§

impl<S> Unpin for SegmentedStore<S>
where S: Unpin, Vec<(<S as Store>::Id, <S as Store>::Item)>: Unpin, Vec<Arc<<S as Store>::Segment>>: Unpin, HashSet<<S as Store>::Id>: Unpin, HashMap<u64, HashSet<<S as Store>::Id>>: Unpin, Option<HashMap<<S as Store>::Id, u64>>: Unpin, Arc<RwLock<Arc<PubState<S>>>>: Unpin,

§

impl<S> UnsafeUnpin for SegmentedStore<S>
where S: UnsafeUnpin, Vec<(<S as Store>::Id, <S as Store>::Item)>: UnsafeUnpin, Vec<Arc<<S as Store>::Segment>>: UnsafeUnpin, HashSet<<S as Store>::Id>: UnsafeUnpin, HashMap<u64, HashSet<<S as Store>::Id>>: UnsafeUnpin, Option<HashMap<<S as Store>::Id, u64>>: UnsafeUnpin, Arc<RwLock<Arc<PubState<S>>>>: UnsafeUnpin,

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.