Skip to main content

Engine

Struct Engine 

Source
pub struct Engine { /* private fields */ }
Expand description

Single-writer KV engine with WAL, segments, compaction, and optional vector/text indexes.

Implementations§

Source§

impl Engine

Source

pub fn export_manifest(&self) -> Result<Vec<SegmentRef>, EdgestoreError>

Returns the local segment manifest as Vec<SegmentRef> for a remote peer to diff against.

Source

pub fn missing_segments(&self, peer_segments: &[SegmentRef]) -> Vec<[u8; 32]>

Returns hashes the peer has that we do not (set diff: peer ∖ local).

Pure computation, no I/O.

Source

pub fn import_segment( &mut self, data: &[u8], hash: &[u8; 32], ) -> Result<ImportResult, EdgestoreError>

Accept raw segment bytes from a peer, verify BLAKE3, write atomically, apply LWW per record.

Returns:

  • Ok(ImportResult::Skipped) if the segment is already present in the local manifest.
  • Ok(ImportResult::HashMismatch) if BLAKE3(data) != claimed hash — segment rejected.
  • Ok(ImportResult::Rejected { reason }) if the segment passes BLAKE3 but is permanently unprocessable (corrupt block, oversized decompressed payload). Do not retry.
  • Ok(ImportResult::Applied { keys_written, keys_skipped }) on success.

// LWW correctness requires NTP synchronization. Clock skew > segment flush interval // can cause incorrect merge outcomes.

Source

pub fn range_merkle_root(&self) -> Result<[u8; 32], EdgestoreError>

Returns the local RangeMerkleTree root for anti-entropy probing.

The root is computed from each segment’s content hash (segment_hash, the BLAKE3 of raw segment bytes). Using segment_hash — rather than the per-segment merkle_root field that is computed differently by SegmentWriter vs. import_segment — ensures that two nodes converge after a successful sync: once B has imported all of A’s segments, their segment_hash sets are identical, so range_merkle_root() returns the same value on both sides.

Algorithm: sort segment hashes lexicographically, then feed them in order through a single BLAKE3 hasher. Returns the all-zero hash when there are no segments.

Source

pub fn compare_merkle( &self, other_root: &[u8; 32], ) -> Result<bool, EdgestoreError>

Returns true if local Merkle root matches other_root (nodes are in sync).

Returns false if diverged — caller should call export_manifest + missing_segments to determine what to pull.

Source§

impl Engine

Source

pub fn build_vector_index(&mut self, ns: &[u8]) -> Result<(), EdgestoreError>

Build an HNSW index for all vectors in the given namespace.

Scans all vector records in __vec__{ns}, builds the graph, serializes it to a sidecar file, and caches it in memory.

Source

pub fn preload_vector_index(&self, ns: &[u8]) -> Result<bool, EdgestoreError>

Preload the HNSW index for a namespace into memory.

Returns true if the index was loaded (or already cached), false if no index exists.

Try an HNSW search, returning None if no fresh index is available.

Used by edgestore-tokio to attempt the fast HNSW path under a read lock before falling back to the paged flat scan. The interior RwLock on the index cache handles lazy loading without needing the outer engine write lock.

Search for the k closest vectors to the query in the given namespace.

Uses HNSW when an index exists and is fresh; falls back to flat scan otherwise.

Source

pub fn vector_search_with_stats( &self, ns: &[u8], query: &VectorRecord, k: usize, metric: Metric, ) -> Result<(Vec<VectorSearchResult>, QueryStats), EdgestoreError>

Vector search with cost accounting. Returns results + QueryStats.

bytes_scanned reflects the sum of encoded vector record sizes examined during a flat scan. For HNSW paths, only the result set bytes are counted (graph traversal does not materialize all vectors).

Source

pub fn vector_page( &self, ns: &[u8], cursor: Option<&[u8]>, page_size: usize, ) -> Result<VectorPage, EdgestoreError>

Fetch one page of decoded vector records for cooperative async flat scans.

Designed for async callers (e.g. edgestore-tokio) that want to iterate through a vector namespace without holding the engine lock for the full flat-scan duration. Takes &self (read lock only) — no HNSW mutation.

Pass None as cursor to start from the beginning. On each call the returned next_key (if Some) is the cursor for the next page.

Source§

impl Engine

Source

pub fn open(config: EdgestoreConfig) -> Result<Engine, EdgestoreError>

Open or create an Engine at the configured path.

Source

pub fn open_readonly(config: EdgestoreConfig) -> Result<Engine, EdgestoreError>

Open an engine in read-only mode.

All write methods (put, delete, vector_put, index_text, etc.) will return Err(EdgestoreError::ReadOnly). Use for replica instances to prevent accidental writes that would cause divergence from the primary.

Source

pub fn with_on_segment_flushed( self, cb: impl Fn(&SegmentMeta) + Send + Sync + 'static, ) -> Self

Register a callback fired after every successful segment flush.

Called from flush_to_segments (both explicit and auto-triggered by put). Receives the new SegmentMeta. Use to wake a replication anti-entropy loop, update external metrics, or trigger downstream processing.

The callback runs synchronously on the calling thread. Keep it fast — e.g. send on a channel, set an atomic flag, or log. Do not call back into the same Engine from within the callback.

Source

pub fn confirmed_lsn(&self) -> u64

Highest LSN whose segment is durably flushed and visible to replicas.

Computed as max(meta.max_lsn) across all segment metas in the manifest. In-flight WAL records that have not yet been flushed to a segment are NOT counted — replicas use this value to decide what is safe to serve.

Returns 0 when no segments have been written yet.

Source

pub fn current_lsn(&self) -> u64

Current LSN counter — the LSN that will be assigned to the next write.

Monotonically increasing. Corresponds to lsn in edgestore.tla. Use confirmed_lsn for the highest LSN durably stored in a segment.

Source

pub fn wal_segment_count(&self) -> usize

Number of WAL segment files currently on disk.

A value above a few dozen usually means flush_to_segments is not being called often enough. Low-write-rate applications should call it periodically (e.g. on a background timer) rather than relying solely on the memtable_max_bytes auto-flush threshold.

Source

pub fn pending_wal_bytes(&self) -> u64

Bytes written to the current (active) WAL file.

Combine with wal_segment_count to estimate total unflushed WAL size. Useful for triggering an explicit flush_to_segments call before the memtable auto-flush threshold is reached.

Source

pub fn write_token(&self) -> u64

Current write token.

A monotonically increasing u64 that survives restarts. When a replica is promoted to primary it increments this token so anti-entropy loops on peers that already observed the old token will recognise the topology change and can refuse or warn on stale segments.

Source

pub fn set_write_token(&mut self, token: u64) -> Result<(), EdgestoreError>

Persist a new write token to disk and update the in-memory value.

Callers must guarantee token > self.write_token() for fencing to hold — ReplicatedEngine::promote_to_primary enforces this invariant.

Source

pub fn vector_count(&self, ns: &[u8]) -> Option<u64>

Returns the number of vectors in the given namespace if the HNSW index is currently loaded in memory, or None if the index has not been loaded.

Call preload_vector_index(ns) first if you need a guaranteed count. This method never triggers a disk scan.

Source

pub fn put( &mut self, ns: &[u8], key: &[u8], val: &[u8], ) -> Result<Lsn, EdgestoreError>

Store a key-value pair in the given namespace.

Source

pub fn put_with_ttl( &mut self, ns: &[u8], key: &[u8], val: &[u8], ttl_secs: u32, ) -> Result<Lsn, EdgestoreError>

Store a key-value pair with a TTL (seconds).

Records expire lazily during compaction based on cohort window.

Source

pub fn get( &self, ns: &[u8], key: &[u8], ) -> Result<Option<Vec<u8>>, EdgestoreError>

Lazy expiry: records inserted with put_with_ttl are returned until compact_once removes their cohort.

Source

pub fn get_with_stats( &self, ns: &[u8], key: &[u8], ) -> Result<(Option<Vec<u8>>, QueryStats), EdgestoreError>

Get with cost accounting. Returns the value (if any) and QueryStats.

stats.segments_scanned is 1 if the key was found in a segment, 0 for a memtable hit or a miss; bytes_scanned is the key+value size of the found entry.

Source

pub fn get_into( &self, ns: &[u8], key: &[u8], buf: &mut Vec<u8>, ) -> Result<bool, EdgestoreError>

Get a value into an existing buffer, avoiding a fresh Vec<u8> allocation.

Returns true if the key was found and buf was filled. Returns false (and leaves buf unchanged) on a miss. Useful for high-throughput callers that reuse a buffer across many lookups.

Source

pub fn delete(&mut self, ns: &[u8], key: &[u8]) -> Result<Lsn, EdgestoreError>

Delete a key in the given namespace (tombstone).

Source

pub fn range( &self, ns: &[u8], start: &[u8], end: &[u8], ) -> Result<Vec<(Vec<u8>, Vec<u8>)>, EdgestoreError>

Lazy expiry: TTL-expired records appear in range results until compaction removes their cohort.

Source

pub fn range_with_stats( &self, ns: &[u8], start: &[u8], end: &[u8], ) -> Result<(Vec<(Vec<u8>, Vec<u8>)>, QueryStats), EdgestoreError>

Range scan with cost accounting. Returns results + QueryStats.

Source

pub fn range_budgeted( &self, ns: &[u8], start: &[u8], end: &[u8], budget: &ScanBudget, ) -> Result<BudgetedScan<(Vec<u8>, Vec<u8>)>, EdgestoreError>

Range scan that stops when the ScanBudget is exhausted. Returns a BudgetedScan that indicates whether the result was truncated.

Source

pub fn prefix( &self, ns: &[u8], prefix: &[u8], ) -> Result<Vec<(Vec<u8>, Vec<u8>)>, EdgestoreError>

Lazy expiry: TTL-expired records appear in prefix results until compaction removes their cohort.

Source

pub fn prefix_with_stats( &self, ns: &[u8], prefix: &[u8], ) -> Result<(Vec<(Vec<u8>, Vec<u8>)>, QueryStats), EdgestoreError>

Prefix scan with cost accounting. Returns results + QueryStats.

Source

pub fn prefix_budgeted( &self, ns: &[u8], prefix: &[u8], budget: &ScanBudget, ) -> Result<BudgetedScan<(Vec<u8>, Vec<u8>)>, EdgestoreError>

Prefix scan that stops when the ScanBudget is exhausted.

Source

pub fn range_page( &self, ns: &[u8], start: &[u8], end: &[u8], cursor: Option<&[u8]>, page_size: usize, ) -> Result<RangePage, EdgestoreError>

Cursor-based forward range page (P4).

Returns up to page_size items in ascending key order starting just after cursor. On the first call pass cursor = None; on subsequent calls pass the next_key returned by the previous call. next_key = None in the result means no more pages.

Each call is bounded at the I/O level: only segments whose key range overlaps the effective [cursor, end) window are read, and reading stops as soon as page_size live items are collected.

Source

pub fn range_rev_page( &self, ns: &[u8], start: &[u8], end: &[u8], cursor: Option<&[u8]>, page_size: usize, ) -> Result<RangePage, EdgestoreError>

Cursor-based reverse range page (P5).

Returns up to page_size items in descending key order starting just below cursor. On the first call pass cursor = None to start from end; on subsequent calls pass the next_key returned by the previous call.

next_key is the smallest key in the current page (the furthest-left point reached). Pass it as cursor to the next call to continue going left. next_key = None means the scan reached start and is exhausted.

Source

pub fn flush_to_segments(&mut self) -> Result<SegmentMeta, EdgestoreError>

Flush the current memtable to a new on-disk segment.

After a successful flush:

  • The memtable is cleared.
  • WAL files older than the current one are deleted (they are now redundant for crash recovery — the segment is the durable record).
  • The on_segment_flushed callback fires if configured.

Returns Err if the memtable is empty (nothing to flush).

This is also called automatically when the memtable exceeds EdgestoreConfig::memtable_max_bytes.

Source

pub fn flush(&mut self) -> Result<(), EdgestoreError>

fsync the current WAL file.

Source

pub fn begin(&mut self) -> Transaction

Start a new multi-record transaction.

Source

pub fn commit_transaction( &mut self, tx: Transaction, ) -> Result<Lsn, EdgestoreError>

Commit a transaction, writing all pending records to the WAL.

Source

pub fn rollback_transaction(&mut self, tx: Transaction)

Roll back a transaction, discarding all pending records.

Source

pub fn compact_once(&mut self) -> Result<CompactionStats, EdgestoreError>

Run one bounded compaction cycle.

Uses wall-clock time to determine which cohorts are expired. Respects the write budget from EdgestoreConfig::compaction_write_budget_bytes. Pinned segments (held by live snapshots) are never removed or rewritten.

After compaction, the segment store is reloaded from disk so subsequent reads see the updated segment list.

Source

pub fn snapshot(&self) -> Result<Snapshot, EdgestoreError>

Return a point-in-time snapshot pinning the current set of segments.

The returned Snapshot holds a reference to the SnapshotRegistry and releases its pins automatically when dropped.

Source

pub fn db_path(&self) -> &Path

Returns the filesystem path to the engine’s database directory.

Used by external crates (e.g. edgestore-repl) to locate segment files.

Source

pub fn metrics(&self) -> MetricsSnapshot

Returns a point-in-time snapshot of all engine metrics.

Source

pub fn list_segment_metas(&self) -> Vec<SegmentMeta>

Return metadata for all live segments.

Used by edgestore-tier to build archived-segment indexes.

Source

pub fn strip_text_index( &mut self, segment_id: u64, ) -> Result<SegmentMeta, EdgestoreError>

Rewrite a segment removing all __text__* namespace records.

After a segment has been archived to cold storage, callers may strip its embedded full-text index records to reclaim local disk space. The archived copy retains the original data; the local copy becomes a compact KV-only segment.

Sets SegmentMeta::text_index_stripped = true on the new segment. Subsequent search_text calls will not find text records from this segment; existing search results are unaffected if the merged __index__ entry was already flushed to a later (unstripped) segment.

Returns Ok(existing_meta) unchanged if the segment has no text records or was already stripped.

§Errors

Returns an error if segment_id is not found, or if the rewrite fails.

Source

pub fn prune_local_segment( &mut self, segment_id: SegmentId, ) -> Result<(), EdgestoreError>

Removes one local segment: deletes its .dat/.idx/.xf/.meta files and its manifest entry. Does not touch any remote/archived copy — for callers that have already confirmed the segment is durably archived elsewhere and just want to reclaim local disk space after a successful archive.

A no-op (returns Ok) if segment_id doesn’t exist locally.

Source

pub fn strip_vector_index( &mut self, segment_id: u64, ) -> Result<SegmentMeta, EdgestoreError>

Strip the embedded vector index from a local segment, rewriting it without __vec__ namespace records. Mirrors Engine::strip_text_index for the vector tier: call this after archive_segments to reclaim local disk space while the full vector data remains available in the remote archive.

Returns the (possibly rewritten) segment metadata. If the segment has no vector records, or if it was already stripped, returns the original metadata unchanged. If ALL records are vector records (no KV or text), the segment is returned unchanged — the caller should decide whether to prune it.

Source

pub fn begin_flush(&mut self) -> Option<FlushWork>

Phase 1 of the 3-phase flush: snapshot the memtable entries and reserve a segment ID. The active memtable is NOT cleared — writes can continue, and reads remain fully served during Phase 2 disk I/O.

Returns None if the memtable is empty. The caller must pass the returned FlushWork::flushed_max_lsn to finalize_flush to drop the snapshotted entries from the memtable after the segment is registered.

Source

pub fn finalize_flush( &mut self, meta: &SegmentMeta, flushed_max_lsn: u64, ) -> Result<(), EdgestoreError>

Phase 3 of the 3-phase flush: register the completed segment, rebuild the memtable keeping only entries written after the snapshot, fire the flush callback, and retire old WAL files.

flushed_max_lsn is FlushWork::flushed_max_lsn from Phase 1. Entries at or below that LSN are in the new segment and are removed from the memtable. Entries above it were written during Phase 2 and are retained.

Trait Implementations§

Source§

impl Drop for Engine

Source§

fn drop(&mut self)

Executes the destructor for this type. Read more
Source§

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)
Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned. Read more
Source§

impl VectorEngine for Engine

Source§

fn vector_put( &mut self, ns: &[u8], key: &[u8], dims: u16, dtype: Dtype, data: &[u8], ) -> Result<Lsn, EdgestoreError>

Store a vector record under the given namespace and key.
Source§

fn vector_get( &self, ns: &[u8], key: &[u8], ) -> Result<Option<VectorRecord>, EdgestoreError>

Retrieve a vector record by namespace and key.
Source§

fn vector_delete( &mut self, ns: &[u8], key: &[u8], ) -> Result<Lsn, EdgestoreError>

Delete a vector record by namespace and key.

Auto Trait Implementations§

§

impl !Freeze for Engine

§

impl !RefUnwindSafe for Engine

§

impl !UnwindSafe for Engine

§

impl Forget for Engine

§

impl Move for Engine

§

impl Send for Engine

§

impl Sync for Engine

§

impl Unpin for Engine

§

impl UnsafeUnpin for Engine

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V