segstore
Generic durable segmented store: write-ahead log, immutable segments, tombstone deletes, checkpoint, and compaction.
A consumer implements the Store trait to say how a batch of items becomes a
segment and how segments merge during compaction; segstore owns the durability
(built on durability) and the LSM-style
lifecycle. The segment representation is opaque, so the same machinery backs an
inverted index (posting-list segments), a graph index (graph-delta segments), or
any other updatable structure.
Example
use ;
use MemoryDirectory;
// A segment is just a batch of (id, item) pairs.
;
let dir = arc;
let mut s = open.unwrap;
s.add.unwrap;
s.add.unwrap;
s.delete.unwrap;
assert!;
Durability
In-memory or on-disk via durability. Each add/delete is logged to a
write-ahead log before it takes effect. A checkpoint writes each new segment to
its own file and atomically publishes a manifest (CRC-checked, with an fsync
barrier on a filesystem backend) naming the current segments and tombstones, then
rotates the WAL (a new epoch-suffixed log is started and the old one deleted) so
the log never grows past one checkpoint interval. Because only new segments are
written, a checkpoint is O(new data), not O(total): the Lucene segments_N /
RocksDB MANIFEST model. Recovery loads the manifest's segment files and replays
only the current epoch's WAL. SyncPolicy::Fsync (via open_with_options) syncs
every WAL record to stable storage; the default flushes to the OS without a
per-op fsync.
Compaction
compact() merges all segments into one and purges tombstones.
compact_tiers() runs size-tiered compaction (Cassandra/Lucene-style): segments
are grouped into size buckets and a full bucket is merged, smallest first, never
exceeding max_merged_len items, with larger segments frozen out so the biggest
one is never rewritten by tiering. Scheduling is the consumer's: call
compact_tiers() when convenient (e.g. a background thread), or set
Options::auto_compact to run it inline after a flush. segment_sizes() and the
CompactionStats returned by both expose the segment-count and merge-cost signal
to watch as the corpus grows.
For bulk ingest (an index build phase), extend(items) syncs the WAL once per
batch instead of per item.
Concurrent reads
Single writer, many readers. reader() returns a cloneable, thread-safe Reader;
reader.view() takes a consistent point-in-time View of the segments and
tombstones that a query holds (lock-free) for its whole duration, even while the
writer adds, deletes, or compacts on another thread. Visibility is commit-style:
a view reflects the state as of the last checkpoint() (which compaction also
performs), so writes since then become visible after the next checkpoint. This is
the Lucene SearcherManager / Tantivy Searcher model, made light by segstore's
in-memory segments (a View is Arc clones; an old segment's memory frees when
the last view holding it drops). Publishing only at the checkpoint keeps the
snapshot off the ingest hot path -- republishing per write would make bulk ingest
quadratic.
Status
0.x; the API and on-disk format may change between minor versions.
Not to be confused with seglog, an append-only event log for event sourcing:
segstore is a mutable index-backing store with deletes and compaction, the layer
above a write-ahead log rather than the log itself.
License
Dual-licensed under MIT or Apache-2.0.