pub struct Engine { /* private fields */ }Expand description
Single-writer KV engine with WAL, segments, compaction, and optional vector/text indexes.
Implementations§
Source§impl Engine
impl Engine
Sourcepub fn export_manifest(&self) -> Result<Vec<SegmentRef>, EdgestoreError>
pub fn export_manifest(&self) -> Result<Vec<SegmentRef>, EdgestoreError>
Returns the local segment manifest as Vec<SegmentRef> for a remote peer to diff against.
Sourcepub fn missing_segments(&self, peer_segments: &[SegmentRef]) -> Vec<[u8; 32]>
pub fn missing_segments(&self, peer_segments: &[SegmentRef]) -> Vec<[u8; 32]>
Returns hashes the peer has that we do not (set diff: peer ∖ local).
Pure computation, no I/O.
Sourcepub fn import_segment(
&mut self,
data: &[u8],
hash: &[u8; 32],
) -> Result<ImportResult, EdgestoreError>
pub fn import_segment( &mut self, data: &[u8], hash: &[u8; 32], ) -> Result<ImportResult, EdgestoreError>
Accept raw segment bytes from a peer, verify BLAKE3, write atomically, apply LWW per record.
Returns:
Ok(ImportResult::Skipped)if the segment is already present in the local manifest.Ok(ImportResult::HashMismatch)if BLAKE3(data) != claimed hash — segment rejected.Ok(ImportResult::Rejected { reason })if the segment passes BLAKE3 but is permanently unprocessable (corrupt block, oversized decompressed payload). Do not retry.Ok(ImportResult::Applied { keys_written, keys_skipped })on success.
// LWW correctness requires NTP synchronization. Clock skew > segment flush interval // can cause incorrect merge outcomes.
Sourcepub fn range_merkle_root(&self) -> Result<[u8; 32], EdgestoreError>
pub fn range_merkle_root(&self) -> Result<[u8; 32], EdgestoreError>
Returns the local RangeMerkleTree root for anti-entropy probing.
The root is computed from each segment’s content hash (segment_hash, the BLAKE3
of raw segment bytes). Using segment_hash — rather than the per-segment
merkle_root field that is computed differently by SegmentWriter vs.
import_segment — ensures that two nodes converge after a successful sync:
once B has imported all of A’s segments, their segment_hash sets are identical,
so range_merkle_root() returns the same value on both sides.
Algorithm: sort segment hashes lexicographically, then feed them in order through a single BLAKE3 hasher. Returns the all-zero hash when there are no segments.
Sourcepub fn compare_merkle(
&self,
other_root: &[u8; 32],
) -> Result<bool, EdgestoreError>
pub fn compare_merkle( &self, other_root: &[u8; 32], ) -> Result<bool, EdgestoreError>
Returns true if local Merkle root matches other_root (nodes are in sync).
Returns false if diverged — caller should call export_manifest + missing_segments to determine what to pull.
Source§impl Engine
impl Engine
Sourcepub fn build_vector_index(&mut self, ns: &[u8]) -> Result<(), EdgestoreError>
pub fn build_vector_index(&mut self, ns: &[u8]) -> Result<(), EdgestoreError>
Build an HNSW index for all vectors in the given namespace.
Scans all vector records in __vec__{ns}, builds the graph,
serializes it to a sidecar file, and caches it in memory.
Sourcepub fn preload_vector_index(&self, ns: &[u8]) -> Result<bool, EdgestoreError>
pub fn preload_vector_index(&self, ns: &[u8]) -> Result<bool, EdgestoreError>
Preload the HNSW index for a namespace into memory.
Returns true if the index was loaded (or already cached), false if no index exists.
Sourcepub fn try_hnsw_search(
&self,
ns: &[u8],
query: &VectorRecord,
k: usize,
) -> Result<Option<Vec<VectorSearchResult>>, EdgestoreError>
pub fn try_hnsw_search( &self, ns: &[u8], query: &VectorRecord, k: usize, ) -> Result<Option<Vec<VectorSearchResult>>, EdgestoreError>
Try an HNSW search, returning None if no fresh index is available.
Used by edgestore-tokio to attempt the fast HNSW path under a read lock
before falling back to the paged flat scan. The interior RwLock on the
index cache handles lazy loading without needing the outer engine write lock.
Sourcepub fn vector_search(
&self,
ns: &[u8],
query: &VectorRecord,
k: usize,
metric: Metric,
) -> Result<Vec<VectorSearchResult>, EdgestoreError>
pub fn vector_search( &self, ns: &[u8], query: &VectorRecord, k: usize, metric: Metric, ) -> Result<Vec<VectorSearchResult>, EdgestoreError>
Search for the k closest vectors to the query in the given namespace.
Uses HNSW when an index exists and is fresh; falls back to flat scan otherwise.
Sourcepub fn vector_search_with_stats(
&self,
ns: &[u8],
query: &VectorRecord,
k: usize,
metric: Metric,
) -> Result<(Vec<VectorSearchResult>, QueryStats), EdgestoreError>
pub fn vector_search_with_stats( &self, ns: &[u8], query: &VectorRecord, k: usize, metric: Metric, ) -> Result<(Vec<VectorSearchResult>, QueryStats), EdgestoreError>
Vector search with cost accounting. Returns results + QueryStats.
bytes_scanned reflects the sum of encoded vector record sizes examined during
a flat scan. For HNSW paths, only the result set bytes are counted (graph
traversal does not materialize all vectors).
Sourcepub fn vector_page(
&self,
ns: &[u8],
cursor: Option<&[u8]>,
page_size: usize,
) -> Result<VectorPage, EdgestoreError>
pub fn vector_page( &self, ns: &[u8], cursor: Option<&[u8]>, page_size: usize, ) -> Result<VectorPage, EdgestoreError>
Fetch one page of decoded vector records for cooperative async flat scans.
Designed for async callers (e.g. edgestore-tokio) that want to iterate
through a vector namespace without holding the engine lock for the full
flat-scan duration. Takes &self (read lock only) — no HNSW mutation.
Pass None as cursor to start from the beginning. On each call the
returned next_key (if Some) is the cursor for the next page.
Source§impl Engine
impl Engine
Sourcepub fn open(config: EdgestoreConfig) -> Result<Engine, EdgestoreError>
pub fn open(config: EdgestoreConfig) -> Result<Engine, EdgestoreError>
Open or create an Engine at the configured path.
Sourcepub fn open_readonly(config: EdgestoreConfig) -> Result<Engine, EdgestoreError>
pub fn open_readonly(config: EdgestoreConfig) -> Result<Engine, EdgestoreError>
Open an engine in read-only mode.
All write methods (put, delete, vector_put, index_text, etc.) will
return Err(EdgestoreError::ReadOnly). Use for replica instances to prevent
accidental writes that would cause divergence from the primary.
Sourcepub fn with_on_segment_flushed(
self,
cb: impl Fn(&SegmentMeta) + Send + Sync + 'static,
) -> Self
pub fn with_on_segment_flushed( self, cb: impl Fn(&SegmentMeta) + Send + Sync + 'static, ) -> Self
Register a callback fired after every successful segment flush.
Called from flush_to_segments (both explicit and auto-triggered by put).
Receives the new SegmentMeta. Use to wake a replication anti-entropy loop,
update external metrics, or trigger downstream processing.
The callback runs synchronously on the calling thread. Keep it fast —
e.g. send on a channel, set an atomic flag, or log. Do not call back
into the same Engine from within the callback.
Sourcepub fn confirmed_lsn(&self) -> u64
pub fn confirmed_lsn(&self) -> u64
Highest LSN whose segment is durably flushed and visible to replicas.
Computed as max(meta.max_lsn) across all segment metas in the manifest.
In-flight WAL records that have not yet been flushed to a segment are NOT
counted — replicas use this value to decide what is safe to serve.
Returns 0 when no segments have been written yet.
Sourcepub fn current_lsn(&self) -> u64
pub fn current_lsn(&self) -> u64
Current LSN counter — the LSN that will be assigned to the next write.
Monotonically increasing. Corresponds to lsn in edgestore.tla. Use
confirmed_lsn for the highest LSN durably stored in a segment.
Sourcepub fn wal_segment_count(&self) -> usize
pub fn wal_segment_count(&self) -> usize
Number of WAL segment files currently on disk.
A value above a few dozen usually means flush_to_segments is not being
called often enough. Low-write-rate applications should call it
periodically (e.g. on a background timer) rather than relying solely on
the memtable_max_bytes auto-flush threshold.
Sourcepub fn pending_wal_bytes(&self) -> u64
pub fn pending_wal_bytes(&self) -> u64
Bytes written to the current (active) WAL file.
Combine with wal_segment_count to estimate total unflushed WAL size.
Useful for triggering an explicit flush_to_segments call before the
memtable auto-flush threshold is reached.
Sourcepub fn write_token(&self) -> u64
pub fn write_token(&self) -> u64
Current write token.
A monotonically increasing u64 that survives restarts. When a replica is promoted to primary it increments this token so anti-entropy loops on peers that already observed the old token will recognise the topology change and can refuse or warn on stale segments.
Sourcepub fn set_write_token(&mut self, token: u64) -> Result<(), EdgestoreError>
pub fn set_write_token(&mut self, token: u64) -> Result<(), EdgestoreError>
Persist a new write token to disk and update the in-memory value.
Callers must guarantee token > self.write_token() for fencing to hold —
ReplicatedEngine::promote_to_primary enforces this invariant.
Sourcepub fn vector_count(&self, ns: &[u8]) -> Option<u64>
pub fn vector_count(&self, ns: &[u8]) -> Option<u64>
Returns the number of vectors in the given namespace if the HNSW index is
currently loaded in memory, or None if the index has not been loaded.
Call preload_vector_index(ns) first if you need a guaranteed count.
This method never triggers a disk scan.
Sourcepub fn put(
&mut self,
ns: &[u8],
key: &[u8],
val: &[u8],
) -> Result<Lsn, EdgestoreError>
pub fn put( &mut self, ns: &[u8], key: &[u8], val: &[u8], ) -> Result<Lsn, EdgestoreError>
Store a key-value pair in the given namespace.
Sourcepub fn put_with_ttl(
&mut self,
ns: &[u8],
key: &[u8],
val: &[u8],
ttl_secs: u32,
) -> Result<Lsn, EdgestoreError>
pub fn put_with_ttl( &mut self, ns: &[u8], key: &[u8], val: &[u8], ttl_secs: u32, ) -> Result<Lsn, EdgestoreError>
Store a key-value pair with a TTL (seconds).
Records expire lazily during compaction based on cohort window.
Sourcepub fn get(
&self,
ns: &[u8],
key: &[u8],
) -> Result<Option<Vec<u8>>, EdgestoreError>
pub fn get( &self, ns: &[u8], key: &[u8], ) -> Result<Option<Vec<u8>>, EdgestoreError>
Lazy expiry: records inserted with put_with_ttl are returned until compact_once removes their cohort.
Sourcepub fn get_with_stats(
&self,
ns: &[u8],
key: &[u8],
) -> Result<(Option<Vec<u8>>, QueryStats), EdgestoreError>
pub fn get_with_stats( &self, ns: &[u8], key: &[u8], ) -> Result<(Option<Vec<u8>>, QueryStats), EdgestoreError>
Get with cost accounting. Returns the value (if any) and QueryStats.
stats.segments_scanned is 1 if the key was found in a segment, 0 for a
memtable hit or a miss; bytes_scanned is the key+value size of the found entry.
Sourcepub fn get_into(
&self,
ns: &[u8],
key: &[u8],
buf: &mut Vec<u8>,
) -> Result<bool, EdgestoreError>
pub fn get_into( &self, ns: &[u8], key: &[u8], buf: &mut Vec<u8>, ) -> Result<bool, EdgestoreError>
Get a value into an existing buffer, avoiding a fresh Vec<u8> allocation.
Returns true if the key was found and buf was filled. Returns false
(and leaves buf unchanged) on a miss. Useful for high-throughput callers
that reuse a buffer across many lookups.
Sourcepub fn delete(&mut self, ns: &[u8], key: &[u8]) -> Result<Lsn, EdgestoreError>
pub fn delete(&mut self, ns: &[u8], key: &[u8]) -> Result<Lsn, EdgestoreError>
Delete a key in the given namespace (tombstone).
Sourcepub fn range(
&self,
ns: &[u8],
start: &[u8],
end: &[u8],
) -> Result<Vec<(Vec<u8>, Vec<u8>)>, EdgestoreError>
pub fn range( &self, ns: &[u8], start: &[u8], end: &[u8], ) -> Result<Vec<(Vec<u8>, Vec<u8>)>, EdgestoreError>
Lazy expiry: TTL-expired records appear in range results until compaction removes their cohort.
Sourcepub fn range_with_stats(
&self,
ns: &[u8],
start: &[u8],
end: &[u8],
) -> Result<(Vec<(Vec<u8>, Vec<u8>)>, QueryStats), EdgestoreError>
pub fn range_with_stats( &self, ns: &[u8], start: &[u8], end: &[u8], ) -> Result<(Vec<(Vec<u8>, Vec<u8>)>, QueryStats), EdgestoreError>
Range scan with cost accounting. Returns results + QueryStats.
Sourcepub fn range_budgeted(
&self,
ns: &[u8],
start: &[u8],
end: &[u8],
budget: &ScanBudget,
) -> Result<BudgetedScan<(Vec<u8>, Vec<u8>)>, EdgestoreError>
pub fn range_budgeted( &self, ns: &[u8], start: &[u8], end: &[u8], budget: &ScanBudget, ) -> Result<BudgetedScan<(Vec<u8>, Vec<u8>)>, EdgestoreError>
Range scan that stops when the ScanBudget is exhausted.
Returns a BudgetedScan that indicates whether the result was truncated.
Sourcepub fn prefix(
&self,
ns: &[u8],
prefix: &[u8],
) -> Result<Vec<(Vec<u8>, Vec<u8>)>, EdgestoreError>
pub fn prefix( &self, ns: &[u8], prefix: &[u8], ) -> Result<Vec<(Vec<u8>, Vec<u8>)>, EdgestoreError>
Lazy expiry: TTL-expired records appear in prefix results until compaction removes their cohort.
Sourcepub fn prefix_with_stats(
&self,
ns: &[u8],
prefix: &[u8],
) -> Result<(Vec<(Vec<u8>, Vec<u8>)>, QueryStats), EdgestoreError>
pub fn prefix_with_stats( &self, ns: &[u8], prefix: &[u8], ) -> Result<(Vec<(Vec<u8>, Vec<u8>)>, QueryStats), EdgestoreError>
Prefix scan with cost accounting. Returns results + QueryStats.
Sourcepub fn prefix_budgeted(
&self,
ns: &[u8],
prefix: &[u8],
budget: &ScanBudget,
) -> Result<BudgetedScan<(Vec<u8>, Vec<u8>)>, EdgestoreError>
pub fn prefix_budgeted( &self, ns: &[u8], prefix: &[u8], budget: &ScanBudget, ) -> Result<BudgetedScan<(Vec<u8>, Vec<u8>)>, EdgestoreError>
Prefix scan that stops when the ScanBudget is exhausted.
Sourcepub fn range_page(
&self,
ns: &[u8],
start: &[u8],
end: &[u8],
cursor: Option<&[u8]>,
page_size: usize,
) -> Result<RangePage, EdgestoreError>
pub fn range_page( &self, ns: &[u8], start: &[u8], end: &[u8], cursor: Option<&[u8]>, page_size: usize, ) -> Result<RangePage, EdgestoreError>
Cursor-based forward range page (P4).
Returns up to page_size items in ascending key order starting just after cursor.
On the first call pass cursor = None; on subsequent calls pass the next_key
returned by the previous call. next_key = None in the result means no more pages.
Each call is bounded at the I/O level: only segments whose key range overlaps the
effective [cursor, end) window are read, and reading stops as soon as page_size
live items are collected.
Sourcepub fn range_rev_page(
&self,
ns: &[u8],
start: &[u8],
end: &[u8],
cursor: Option<&[u8]>,
page_size: usize,
) -> Result<RangePage, EdgestoreError>
pub fn range_rev_page( &self, ns: &[u8], start: &[u8], end: &[u8], cursor: Option<&[u8]>, page_size: usize, ) -> Result<RangePage, EdgestoreError>
Cursor-based reverse range page (P5).
Returns up to page_size items in descending key order starting just below
cursor. On the first call pass cursor = None to start from end; on subsequent
calls pass the next_key returned by the previous call.
next_key is the smallest key in the current page (the furthest-left point
reached). Pass it as cursor to the next call to continue going left.
next_key = None means the scan reached start and is exhausted.
Sourcepub fn flush_to_segments(&mut self) -> Result<SegmentMeta, EdgestoreError>
pub fn flush_to_segments(&mut self) -> Result<SegmentMeta, EdgestoreError>
Flush the current memtable to a new on-disk segment.
After a successful flush:
- The memtable is cleared.
- WAL files older than the current one are deleted (they are now redundant for crash recovery — the segment is the durable record).
- The
on_segment_flushedcallback fires if configured.
Returns Err if the memtable is empty (nothing to flush).
This is also called automatically when the memtable exceeds
EdgestoreConfig::memtable_max_bytes.
Sourcepub fn flush(&mut self) -> Result<(), EdgestoreError>
pub fn flush(&mut self) -> Result<(), EdgestoreError>
fsync the current WAL file.
Sourcepub fn begin(&mut self) -> Transaction
pub fn begin(&mut self) -> Transaction
Start a new multi-record transaction.
Sourcepub fn commit_transaction(
&mut self,
tx: Transaction,
) -> Result<Lsn, EdgestoreError>
pub fn commit_transaction( &mut self, tx: Transaction, ) -> Result<Lsn, EdgestoreError>
Commit a transaction, writing all pending records to the WAL.
Sourcepub fn rollback_transaction(&mut self, tx: Transaction)
pub fn rollback_transaction(&mut self, tx: Transaction)
Roll back a transaction, discarding all pending records.
Sourcepub fn compact_once(&mut self) -> Result<CompactionStats, EdgestoreError>
pub fn compact_once(&mut self) -> Result<CompactionStats, EdgestoreError>
Run one bounded compaction cycle.
Uses wall-clock time to determine which cohorts are expired.
Respects the write budget from EdgestoreConfig::compaction_write_budget_bytes.
Pinned segments (held by live snapshots) are never removed or rewritten.
After compaction, the segment store is reloaded from disk so subsequent reads see the updated segment list.
Sourcepub fn snapshot(&self) -> Result<Snapshot, EdgestoreError>
pub fn snapshot(&self) -> Result<Snapshot, EdgestoreError>
Return a point-in-time snapshot pinning the current set of segments.
The returned Snapshot holds a reference to the SnapshotRegistry and
releases its pins automatically when dropped.
Sourcepub fn db_path(&self) -> &Path
pub fn db_path(&self) -> &Path
Returns the filesystem path to the engine’s database directory.
Used by external crates (e.g. edgestore-repl) to locate segment files.
Sourcepub fn metrics(&self) -> MetricsSnapshot
pub fn metrics(&self) -> MetricsSnapshot
Returns a point-in-time snapshot of all engine metrics.
Sourcepub fn list_segment_metas(&self) -> Vec<SegmentMeta>
pub fn list_segment_metas(&self) -> Vec<SegmentMeta>
Return metadata for all live segments.
Used by edgestore-tier to build archived-segment indexes.
Sourcepub fn strip_text_index(
&mut self,
segment_id: u64,
) -> Result<SegmentMeta, EdgestoreError>
pub fn strip_text_index( &mut self, segment_id: u64, ) -> Result<SegmentMeta, EdgestoreError>
Rewrite a segment removing all __text__* namespace records.
After a segment has been archived to cold storage, callers may strip its embedded full-text index records to reclaim local disk space. The archived copy retains the original data; the local copy becomes a compact KV-only segment.
Sets SegmentMeta::text_index_stripped = true on the new segment. Subsequent
search_text calls will not find text records from this segment; existing
search results are unaffected if the merged __index__ entry was already flushed
to a later (unstripped) segment.
Returns Ok(existing_meta) unchanged if the segment has no text records or was
already stripped.
§Errors
Returns an error if segment_id is not found, or if the rewrite fails.
Sourcepub fn prune_local_segment(
&mut self,
segment_id: SegmentId,
) -> Result<(), EdgestoreError>
pub fn prune_local_segment( &mut self, segment_id: SegmentId, ) -> Result<(), EdgestoreError>
Removes one local segment: deletes its .dat/.idx/.xf/.meta files and
its manifest entry. Does not touch any remote/archived copy — for callers
that have already confirmed the segment is durably archived elsewhere and
just want to reclaim local disk space after a successful archive.
A no-op (returns Ok) if segment_id doesn’t exist locally.
Sourcepub fn strip_vector_index(
&mut self,
segment_id: u64,
) -> Result<SegmentMeta, EdgestoreError>
pub fn strip_vector_index( &mut self, segment_id: u64, ) -> Result<SegmentMeta, EdgestoreError>
Strip the embedded vector index from a local segment, rewriting it without
__vec__ namespace records. Mirrors Engine::strip_text_index for the vector
tier: call this after archive_segments to reclaim local disk space while the
full vector data remains available in the remote archive.
Returns the (possibly rewritten) segment metadata. If the segment has no vector records, or if it was already stripped, returns the original metadata unchanged. If ALL records are vector records (no KV or text), the segment is returned unchanged — the caller should decide whether to prune it.
Sourcepub fn begin_flush(&mut self) -> Option<FlushWork>
pub fn begin_flush(&mut self) -> Option<FlushWork>
Phase 1 of the 3-phase flush: snapshot the memtable entries and reserve a segment ID. The active memtable is NOT cleared — writes can continue, and reads remain fully served during Phase 2 disk I/O.
Returns None if the memtable is empty. The caller must pass the returned
FlushWork::flushed_max_lsn to finalize_flush to drop the snapshotted
entries from the memtable after the segment is registered.
Sourcepub fn finalize_flush(
&mut self,
meta: &SegmentMeta,
flushed_max_lsn: u64,
) -> Result<(), EdgestoreError>
pub fn finalize_flush( &mut self, meta: &SegmentMeta, flushed_max_lsn: u64, ) -> Result<(), EdgestoreError>
Phase 3 of the 3-phase flush: register the completed segment, rebuild the memtable keeping only entries written after the snapshot, fire the flush callback, and retire old WAL files.
flushed_max_lsn is FlushWork::flushed_max_lsn from Phase 1. Entries
at or below that LSN are in the new segment and are removed from the memtable.
Entries above it were written during Phase 2 and are retained.