pub struct SegmentedStore<S: Store> { /* private fields */ }Expand description
A generic, durable, segmented mutable store.
Implementations§
Source§impl<S: Store> SegmentedStore<S>
impl<S: Store> SegmentedStore<S>
Sourcepub fn open(
dir: Arc<dyn Directory>,
store: S,
flush_threshold: usize,
) -> PersistenceResult<Self>
pub fn open( dir: Arc<dyn Directory>, store: S, flush_threshold: usize, ) -> PersistenceResult<Self>
Open (or create) a store backed by dir, recovering any prior state from
the checkpoint plus the write-ahead log.
flush_threshold is the buffered-add count that triggers a new segment.
Sourcepub fn open_with_options(
dir: Arc<dyn Directory>,
store: S,
opts: Options,
) -> PersistenceResult<Self>
pub fn open_with_options( dir: Arc<dyn Directory>, store: S, opts: Options, ) -> PersistenceResult<Self>
Like Self::open, but with full Options (per-write durability,
size-tiered compaction tuning, auto-compaction).
SyncPolicy::Fsync requires a filesystem-backed Directory; opening
with it on a backend that lacks file_path() (e.g. an in-memory
directory) returns an error rather than silently degrading to a flush.
Sourcepub fn reader(&self) -> Reader<S>
pub fn reader(&self) -> Reader<S>
A cloneable handle for concurrent snapshot reads while this writer mutates.
Clone it across threads; each Reader::view is a consistent snapshot.
Sourcepub fn add(&mut self, id: S::Id, item: S::Item) -> PersistenceResult<()>
pub fn add(&mut self, id: S::Id, item: S::Item) -> PersistenceResult<()>
Add (or re-add) an item. Durably logged before it becomes visible.
A re-add replaces the id’s previous value (last write wins), including a
value deleted earlier. The buffered copy is replaced in place. A copy
already sealed in a segment stays in that segment’s payload until a merge
drops it, but is marked superseded: Self::is_live_in,
View::is_live_in, and SegmentCatalog::is_live_in report it dead,
so a consumer that filters each segment with them sees only the newest
copy. Tracking sealed copies needs Store::item_ids (implemented by
DefaultStore); without it only the buffered copy is replaced, and a
sealed copy stays live next to the new one.
Sourcepub fn extend(
&mut self,
items: impl IntoIterator<Item = (S::Id, S::Item)>,
) -> PersistenceResult<()>
pub fn extend( &mut self, items: impl IntoIterator<Item = (S::Id, S::Item)>, ) -> PersistenceResult<()>
Add (or re-add) many items, syncing the WAL once for the whole batch instead of per item. This is the bulk-ingest path (the build phase of an index, e.g. lexir/sporse loading a corpus): per-item flush is the dominant cost there, and one sync per batch is several times faster on a real disk.
The durability boundary is the batch’s final sync, so a crash mid-batch may leave a prefix of the batch durable (each item is an independently CRC-checked WAL record); recovery yields a consistent prefix, never a half-written item. Auto-compaction (if enabled) still runs when a segment seals during the batch.
Sourcepub fn delete(&mut self, id: S::Id) -> PersistenceResult<()>
pub fn delete(&mut self, id: S::Id) -> PersistenceResult<()>
Tombstone an item. Durably logged before it takes effect.
Sourcepub fn compact(&mut self) -> PersistenceResult<CompactionStats>
pub fn compact(&mut self) -> PersistenceResult<CompactionStats>
Merge ALL segments into one, dropping tombstoned ids and purging the tombstone set, then checkpoint. This is the full compaction; it is the only path that touches frozen (over-cap) segments and the only one that purges tombstones (a partial tier merge cannot, since a tombstoned id may live in a segment it did not merge).
Sourcepub fn compact_tiers(&mut self) -> PersistenceResult<CompactionStats>
pub fn compact_tiers(&mut self) -> PersistenceResult<CompactionStats>
Run all currently-eligible size-tier merges (size-tiered compaction) per the
TierConfig, persisting the result if anything merged. Segments are grouped
into size buckets (a Cassandra-style bucket_low .. bucket_high band around a
running average, with a Options::flush_threshold floor); a bucket with at
least min_merge segments is merged (smallest first, up to max_merge at once,
never exceeding max_merged_len items), dropping tombstoned ids from the merged
output. Segments above max_merged_len / 2 are frozen out, so the largest segment
is never rewritten here. The global tombstone set is NOT purged (only Self::compact
can); a tombstoned id surviving in a frozen segment stays filtered by Self::is_live.
Scheduling is the consumer’s: call this when convenient (e.g. on a background
thread) so merge latency stays off the ingest path, or set Options::auto_compact.
Sourcepub fn force_merge_to(
&mut self,
max_segments: usize,
) -> PersistenceResult<CompactionStats>
pub fn force_merge_to( &mut self, max_segments: usize, ) -> PersistenceResult<CompactionStats>
Consolidate on demand: merge segments until at most max_segments remain,
ignoring the size band, the per-bucket minimum, and the TierConfig cap
(this is explicit user intent, e.g. before a read-heavy phase, so it may
produce a segment larger than max_merged_len). Merging to a single segment
(max_segments <= 1) also purges the tombstone set, exactly like
Self::compact; merging to more keeps the set (an id may live in a segment
the call did not merge). Smallest segments are merged first to keep the work
down. A no-op when already at or below max_segments.
Sourcepub fn has_eligible_tier(&self) -> bool
pub fn has_eligible_tier(&self) -> bool
Whether at least one size bucket is eligible to merge right now.
Sourcepub fn stored_len(&self) -> usize
pub fn stored_len(&self) -> usize
Total items physically stored across segments and the buffer. This counts
tombstoned-but-not-yet-purged items (segstore cannot see inside an opaque
segment to subtract them); pair with Self::tombstone_count to gauge
space amplification.
Sourcepub fn space_amplification(&self) -> Option<f64>
pub fn space_amplification(&self) -> Option<f64>
Space amplification = stored items / live items, or None if any segment’s
Store::live_len returns None (the default). A value near 1.0 means
little dead data; a higher value means tombstoned/obsolete items are
accumulating, reclaimable with Self::reclaim_tombstones or Self::compact.
Sourcepub fn reclaim_tombstones(
&mut self,
min_live_ratio: f64,
) -> PersistenceResult<CompactionStats>
pub fn reclaim_tombstones( &mut self, min_live_ratio: f64, ) -> PersistenceResult<CompactionStats>
Merge segments whose live ratio (live_len / segment_len) is below
min_live_ratio, reclaiming their dead (tombstoned) entries into one fresh
segment. The cheap alternative to Self::compact when only a few segments
are tombstone-heavy. Requires Store::live_len; a no-op if it returns
None or nothing qualifies. Keeps the tombstone set (a reclaimed id may
still live in a segment this did not touch); only compact purges it.
Sourcepub fn segment_sizes(&self) -> Vec<usize>
pub fn segment_sizes(&self) -> Vec<usize>
Per-segment item counts, oldest segment first. The size distribution the consumer needs to watch the crossover signal (segment count + per-query fan-out vs corpus size) that says a multi-segment store is outgrowing the search-all-segments model.
Sourcepub fn checkpoint(&mut self) -> PersistenceResult<()>
pub fn checkpoint(&mut self) -> PersistenceResult<()>
Snapshot the current segments + tombstones durably, then rotate the WAL: start a fresh epoch and delete the old log so it cannot grow unbounded.
The checkpoint is the commit point. It records the new epoch, so once it is durable, recovery replays only the new (initially empty) WAL; the old WAL is superseded. A crash after the checkpoint but before the old WAL is deleted is safe: recovery never reads the old epoch, and the stale file is GC’d on the next open.
If the manifest write fails, the new manifest may already be visible, so
the store is poisoned: later writes and checkpoints return InvalidState.
Reopen the store to recover.
Sourcepub fn segments(&self) -> &[Arc<S::Segment>]
pub fn segments(&self) -> &[Arc<S::Segment>]
The immutable segments, oldest first (each derefs to S::Segment). Query
these plus Self::buffer, filtering segment copies with Self::is_live_in. Segments are
Arc-shared: an unchanged segment keeps its identity across mutations, so a
consumer can cache per-segment state keyed by Arc::as_ptr and rebuild only
new segments. For a consistent view while another thread mutates, use a
Reader / View instead. For a cache key that also survives a restart,
use Self::segment_ids.
Sourcepub fn segment_ids(&self) -> &[u64]
pub fn segment_ids(&self) -> &[u64]
The stable persistence id of each segment in Self::segments, same order.
Unlike the segment’s Arc pointer (stable only within one process), this id
names the segment’s segstore.seg.<id> file and is stable across checkpoints
and restarts, so a consumer can key a persisted per-segment built-index
cache on it and reload across a process restart instead of rebuilding from the
raw payload. An id is never reused after its segment is compacted away.
Sourcepub fn dir(&self) -> &Arc<dyn Directory> ⓘ
pub fn dir(&self) -> &Arc<dyn Directory> ⓘ
The directory this store is backed by, so a consumer can persist a built
per-segment index alongside the segments. Write the index into the reserved
sidecar name from Self::index_name, keyed by a stable Self::segment_ids
id; segstore garbage-collects it when its segment is compacted away (on the
same crash-safe schedule as the segment files). A missing or stale sidecar
means “rebuild from the raw segment.” This is the persist-the-built-index
hook: recovery loads the sidecar instead of rebuilding every consumer index.
Sourcepub fn try_index_name(&self, id: u64, kind: &str) -> PersistenceResult<String>
pub fn try_index_name(&self, id: u64, kind: &str) -> PersistenceResult<String>
The reserved sidecar file name for a per-segment index of the given kind
(e.g. "hnsw"), keyed by a stable Self::segment_ids id. kind must be
non-empty ASCII alphanumeric plus _ or -. It is a short namespace tag,
not a compatibility proof; put full algorithm/config/input fingerprints in
the sidecar bytes and rebuild from the raw segment when they mismatch. Use
with Self::dir to persist/load a built index that segstore GCs in
lockstep with the segment. Reader/searcher code that only has a View
can use the module-level try_index_name helper with View::segment_ids.
Sourcepub fn index_name(&self, id: u64, kind: &str) -> String
pub fn index_name(&self, id: u64, kind: &str) -> String
Infallible convenience wrapper around Self::try_index_name.
Panics if kind violates the documented sidecar-kind grammar.
Sourcepub fn is_live(&self, id: &S::Id) -> bool
pub fn is_live(&self, id: &S::Id) -> bool
Whether id is not tombstoned.
This is per-id: when a re-added id has copies in several segments, it is
true for all of them. Filter a segment’s contents with
Self::is_live_in instead. Every buffered item is live.
Sourcepub fn is_live_in(&self, segment_id: u64, id: &S::Id) -> bool
pub fn is_live_in(&self, segment_id: u64, id: &S::Id) -> bool
Whether the copy of id in segment segment_id (an entry of
Self::segment_ids) is live: id is not tombstoned and was not
re-added after that segment sealed. At most one copy of an id passes
this check (see Self::add).
Sourcepub fn live_segment_of(&self, id: &S::Id) -> Option<u64>
pub fn live_segment_of(&self, id: &S::Id) -> Option<u64>
The id of the segment holding id’s live sealed copy, or None if id
is buffered, deleted, absent, or the store does not implement
Store::item_ids.
A re-add or delete of id changes that segment’s live set, so a consumer
that caches a per-segment index can call this first and drop just that
segment’s cache entry, without scanning segments for the id.
Sourcepub fn live_ids(&self, segment_id: u64) -> Option<LiveIds<S::Id>>
pub fn live_ids(&self, segment_id: u64) -> Option<LiveIds<S::Id>>
Canonical live-id set of one sealed segment: the segment’s ids filtered
by Self::is_live_in, sorted and deduped.
This is the fingerprint a sidecar cache keys on. Returns None if
segment_id is not a current segment or the store’s Store
implementation does not list ids (Store::item_ids). Buffered adds are
not part of any segment’s live-id set.
Sourcepub fn segment_count(&self) -> usize
pub fn segment_count(&self) -> usize
Number of immutable segments.
Sourcepub fn tombstone_count(&self) -> usize
pub fn tombstone_count(&self) -> usize
Number of tombstoned ids.
Trait Implementations§
Source§impl<S: Store> SidecarHost for SegmentedStore<S>
impl<S: Store> SidecarHost for SegmentedStore<S>
Source§fn try_sidecar_name(
&self,
segment_id: u64,
kind: &str,
) -> PersistenceResult<String>
fn try_sidecar_name( &self, segment_id: u64, kind: &str, ) -> PersistenceResult<String>
segment_id, validated against this host’s
known segment set where the type can check it.Source§fn sidecar_name(&self, segment_id: u64, kind: &str) -> String
fn sidecar_name(&self, segment_id: u64, kind: &str) -> String
Self::try_sidecar_name. Read moreSource§fn sidecar_file_path(
&self,
segment_id: u64,
kind: &str,
) -> PersistenceResult<Option<PathBuf>>
fn sidecar_file_path( &self, segment_id: u64, kind: &str, ) -> PersistenceResult<Option<PathBuf>>
Source§fn sidecar_exists(&self, name: &str) -> bool
fn sidecar_exists(&self, name: &str) -> bool
Source§fn read_sidecar(&self, name: &str) -> PersistenceResult<Option<Vec<u8>>>
fn read_sidecar(&self, name: &str) -> PersistenceResult<Option<Vec<u8>>>
None if it is absent.Source§fn write_sidecar(&self, name: &str, bytes: &[u8]) -> PersistenceResult<()>
fn write_sidecar(&self, name: &str, bytes: &[u8]) -> PersistenceResult<()>
Source§fn delete_sidecar(&self, name: &str) -> PersistenceResult<()>
fn delete_sidecar(&self, name: &str) -> PersistenceResult<()>
Directory::delete.