Struct RuleEngine 

pub struct RuleEngine { /* private fields */ }

Implementations

impl RuleEngine

pub fn new() -> Self

pub fn reset_chain_state(&mut self)

Reset the transient chaining state.

Called by db.rs from the same RAII guard that restores emit_deltas after every apply, so a panic unwinding out of a hook cannot leave chain_depth non-zero — which would make begin_chain compute active = false and silently disable chaining for the life of the engine. On the normal path this state is already clean and the call is a no-op.

pub fn rules(&self) -> impl Iterator<Item = &RuleDef>

pub fn chain_truncations(&self) -> u64

How many writes hit MAX_CHAIN_DEPTH with rule-relevant work still pending, since this engine was constructed. A non-zero value means some derived edges beyond the cap are stale: the store is not a fixpoint of its own rule set, and no single later write will repair it. Not persisted; replay re-runs the same hooks, so the value is re-derived identically on reopen.

pub fn is_owned(&self, etype: u32, src: u32, dst: u32) -> bool

pub fn provenance(&self) -> &BTreeMap<String, BTreeSet<(u32, u32, u32)>>

Read-only view of the provenance map: rule name → set of (etype_sym, src, dst).

Triggers a one-time lazy decode from retained bytes when called before the first mutation on a clean-open (no-WAL) store.

pub fn provenance_touching( &self, node: u32, ) -> impl Iterator<Item = (&str, u32, u32, u32)> + '_

O(degree) reverse-index lookup: every provenance triple that touches node.

Dispatches to the lazy-decoded or live index depending on whether retained bytes have already been consumed by a mutation.

pub fn provenance_touching_len(&self, node: u32) -> usize

Number of provenance triples incident on node.

pub fn is_tripped(&self, name: &str) -> bool

One-way latch: true after a budget breach until Self::rebuild is the only exit (and only if the full desired set then fits).

pub fn fire_count(&self, name: &str) -> u64

Evaluations of this rule: one tick per on_node_changed fire, and one tick per participating node on backfill and rebuild (even when rebuild is a provenance no-op).

pub fn drain_deltas(&mut self) -> Vec<EngineEdgeDelta>

Drain and return all pending edge-fire / retract deltas since the last call. Callers (db.rs log_then_apply_with) invoke this after a successful WAL commit + apply to build [DbEvent]s for live subscriptions. [GraphDb::open_with] drains and discards after WAL replay so replay noise never leaks to subscribers.

T2 note (as-of replay)

When Plan-15 T2 adds as-of replay for subscribers, that path should call apply-only (no log_then_apply_with) and then call drain_deltas() to feed those events to the replaying subscriber. The suppression is already in place: apply accumulates but never emits; drain_deltas is the only emission gate.

pub fn pending_delta_count(&self) -> usize

Number of accumulated deltas not yet drained. Used by debug_assert in log_then_apply_with to catch stale-delta bugs.

pub fn pending_deltas_since(&self, cursor: usize) -> &[EngineEdgeDelta]

Borrow the slice of deltas accumulated since cursor without consuming them. cursor should be the value returned by pending_delta_count() before an engine call.

The returned slice is valid until the next call to drain_deltas(). T1’s drain discipline is preserved: these deltas are still in the buffer and will be drained by log_then_apply_with after apply returns.

pub fn to_persist( &self, ) -> (Vec<RuleDef>, BTreeMap<String, BTreeSet<(u32, u32, u32)>>, BTreeMap<String, bool>, BTreeMap<String, u64>)

Snapshot support: definitions + provenance + tripped/fires. Candidate indexes and the by_node reverse index are NOT included (derived: reindex_all / rebuild_by_node on open).

pub fn from_persist( rules: Vec<RuleDef>, prov: BTreeMap<String, BTreeSet<(u32, u32, u32)>>, tripped: BTreeMap<String, bool>, fires: BTreeMap<String, u64>, ) -> Self

Reconstruct engine from a snapshot. Caller must call reindex_all after.

pub fn set_emit_deltas(&mut self, emit: bool)

Enable or disable delta accumulation.

Set to true before the first subscriber or view is added. Set to false when the last subscriber and last view are removed. See the emit_deltas field doc for the safety invariant.

pub fn emit_deltas(&self) -> bool

Whether delta accumulation is currently enabled.

pub fn take_rebuild_needed(&mut self) -> Vec<String>

Drain rule names that exceeded the IVF dst-drift rebuild threshold during the most recent on_node_changed / on_node_removed.

pub fn queue_rebuild_needed(&mut self, name: String)

Re-queue name so a later write can issue RebuildRule.

Used when auto-rebuild WAL IO fails after a durable user op.

pub fn builds_in_progress(&self) -> Vec<BuildProgress>

Rules whose vector index is still being built, in name order.

Empty for every store whose rules were created over a corpus at or below crate::HNSW_BUILD_BATCH vectors — that is, for every store that behaved the way 0.6.5 behaved.

pub fn pump_index_build(&mut self, g: &mut GraphMut<'_>) -> Vec<BuildProgress>

Advance every pending build by one slice.

Returns the builds that finished, in rule order, carrying their final indexed/total so a caller can report what it just completed. The caller must backfill each of them through RebuildRule: the rule derives nothing until it does, and it has already vanished from RuleEngine::builds_in_progress.

Does at most crate::HNSW_BUILD_BATCH vector inserts per pending rule, so a caller can drive a large build to completion without holding a write lock for more than a slice at a time.

pub fn export_ivf_state(&self) -> BTreeMap<String, RuleIvfExport>

Export IVF state for all approximate rules. Passed to snapshot() in core-api and stored in the V4 snapshot so open() can restore cluster assignments without re-fitting k-means.

pub fn reindex_all( &mut self, ids: &IdMap, syms: &Interner, labels: &[u32], props: ColumnsView<'_>, )

Rebuild all candidate indexes by scanning every node. Call on open.

pub fn reindex_all_load_ivf( &mut self, ids: &IdMap, syms: &Interner, labels: &[u32], props: ColumnsView<'_>, ivf_state: BTreeMap<String, RuleIvfExport>, )

Like reindex_all but LOADS persisted IVF state for approximate rules instead of re-fitting k-means. This eliminates the cold-start re-fit cost when opening a V4 snapshot.

ivf_state: map from rule name to (src_export, dst_export) as produced by export_ivf_state / stored in the V4 snapshot.

For approximate rules absent from ivf_state (e.g. a rule added after the snapshot), falls back to fit_ivf_clusters.

Always rebuilds every approximate rule’s HNSW graph from scratch, at a cost superlinear in the number of embeddings. Nothing in this repository calls it; it is retained only because it is published API. Any caller holding persisted HNSW blobs — every open path does — must use RuleEngine::reindex_all_load_state, which installs those graphs and skips the build instead of doing it and throwing it away.

pub fn reindex_all_load_state( &mut self, ids: &IdMap, syms: &Interner, labels: &[u32], props: ColumnsView<'_>, ivf_state: BTreeMap<String, RuleIvfExport>, hnsw_state: BTreeMap<String, (Vec<u8>, Vec<u8>)>, )

Like reindex_all_load_ivf, but also restores the persisted HNSW graphs instead of rebuilding them.

hnsw_state: map from rule name to (src_blob, dst_blob) as produced by export_hnsw_state / stored in the snapshot.

The persisted graph is adopted before the node scan, and the scan is told which ids it already holds so it inserts only what the snapshot did not carry. That is the whole point — building the graph during the scan is superlinear in the number of embeddings, and the persisted graph replaced it wholesale anyway, so the build was pure waste on every open. Adopting first also means a node the scan does see but the graph does not — a rule whose blob predates a write — is inserted rather than dropped. An incomplete blob already registered in pending_builds is the exception: the scan files every non-vector leg and leaves the HNSW remainder to RuleEngine::pump_index_build.

A side falls back to the full rebuild when:

• hnsw_state has no entry for the rule — a store written before HNSW persistence existed, or a rule created since the last snapshot;
• the blob for that side is empty — the side had no graph to export;
• the blob is corrupt or carries a version this build does not read — the reason is logged to stderr and the scan rebuilds the graph.

Entries naming a rule this engine does not treat as approximate are left for load_hnsw_state after the scan, exactly as before.

pub fn store_snapshot_state( &self, hnsw_blobs: BTreeMap<String, (Vec<u8>, Vec<u8>)>, ivf_bytes: Vec<u8>, node_count: u32, )

Store HNSW blobs and raw IVF bytes from a snapshot without deserializing.

Called from restore_snapshot_state in db.rs. Neither the HNSW graphs nor the IVF centroids are materialized here; they are consumed lazily:

• consume_retained_state_eager (WAL-present open, before WAL replay)
• The mutation-hook lazy-init guard (clean open, first-write cost)
• ensure_hnsw_loaded (first ANN query on a clean open)

node_count is the number of id slots the snapshot holds. It is the line between “the snapshot had this node” and “this node is newer”, which reindex_all_load_state needs to recognise an interrupted build without mistaking an in-flight write for one.

pub fn register_incomplete_hnsw_builds( &mut self, extra: &BTreeMap<String, (Vec<u8>, Vec<u8>)>, g: &GraphMut<'_>, )

Register a sliced build a snapshot cut short, from blobs whose complete flag is false.

pending_builds is not persisted; the blob flag is. A clean open never runs the node scan, so this is how serve’s ticker learns there is work without waiting for a write.

extra is the incomplete (src, dst) pair per rule, typically peeked from a V8 mmap without copying complete graphs. When it is empty, the retained blobs from Self::store_snapshot_state are inspected instead (V5–V7, or a caller that already loaded the section).

No-op when indexes are already populated: the scan’s cut_short path owns that case and uses the same [PendingBuild] representation.

pub fn store_provenance_bytes(&self, bytes: Vec<u8>)

Store raw rkyv provenance bytes retained from a V8 snapshot.

Called from restore_v8_base in db.rs after open. Provenance is not decoded here; it is materialized lazily — either by the &self read path (ensure_provenance_loaded) for stats/explain, or by the &mut self write path (ensure_provenance_loaded_mut) on the first mutation.

pub fn ensure_provenance_loaded(&self)

Populate lazy_provenance from retained bytes for &self read paths.

Uses OnceLock for exactly-once initialization. The retained bytes are NOT consumed here; ensure_provenance_loaded_mut still has access to them for the write path. After the first mutation, retained_provenance_bytes is None and callers switch to the live self.provenance field instead.

pub fn ensure_provenance_loaded_mut(&mut self)

Decode and install retained provenance bytes into the live mutable fields.

No-op if bytes have already been consumed or were never stored. Must be called under &mut self before any operation that reads or diffs against self.provenance, self.owned, or self.by_node.

pub fn consume_retained_state_eager( &mut self, ids: &IdMap, syms: &Interner, labels: &[u32], props: ColumnsView<'_>, )

Eagerly consume retained snapshot state before WAL replay.

Call this in open_with when the WAL has records. Runs the O(n) node scan + restores persisted IVF centroids and HNSW blobs so that WAL replay finds fully-populated indexes. Marks indexes_populated = true.

pub fn ensure_hnsw_loaded(&self)

Deserialize retained HNSW blobs into lazy_hnsw for the clean-open ANN read path. Takes &self so it can be called from find_similar_vector and search_hybrid under a shared (db.read()) lock.

Uses OnceLock to guarantee exactly-once initialization even under concurrent shared access. The retained blobs are borrowed (not consumed) so that a subsequent first-mutation call to consume_retained_state_eager can still load the persisted HNSW graphs into self.indexes.

Called before the first ANN query on a clean-open (no WAL) store.

pub fn needs_index_population(&self) -> bool

True when a write has to populate the candidate indexes before it mutates anything.

The lazy population is a full node scan, and it reads the graph it is handed. Run from inside a hook it therefore reads the half-applied record — the in-flight node’s new label and props are already in the columns — and the scan then attributes that node’s vector to the snapshot, which is how a perfectly ordinary write came to look like a build the store had been killed in the middle of. Hoisting it to before the mutation makes the scan see exactly the persisted state, and the write’s own hook then inserts its vector through the normal path.

pub fn populate_indexes(&mut self, g: &GraphMut<'_>)

Build every rule’s candidate index from g if that has not happened yet. The &mut self entry point behind RuleEngine::needs_index_population.

pub fn indexes_populated(&self) -> bool

Returns true if candidate indexes have been built (either eagerly or via the lazy mutation-hook trigger).

pub fn export_hnsw_state(&self) -> BTreeMap<String, (Vec<u8>, Vec<u8>)>

Export the HNSW graph of every rule with a vector leg as an opaque bincoded blob.

Returns a map from rule name to (src_blob, dst_blob). An empty Vec means the corresponding side has no initialized HNSW graph.

pub fn export_hnsw_state_passthrough( &self, ) -> BTreeMap<String, (Vec<u8>, Vec<u8>)>

Returns HNSW state for snapshotting. When indexes are not yet populated (clean open with no mutation), returns the retained raw blobs directly so that a migrate/snapshot does not silently drop fitted indexes.

pub fn retained_ivf_bytes_clone(&self) -> Option<Vec<u8>>

Returns a clone of the retained raw IVF bincode bytes.

Returns None if no bytes are retained (fresh store or indexes already consumed by a mutation). Used by snapshot_with for passthrough when indexes have not yet been populated.

pub fn load_hnsw_state(&mut self, blobs: BTreeMap<String, (Vec<u8>, Vec<u8>)>)

Restore HNSW graphs from bincoded blobs (overrides any graphs the node scan built).

The open paths no longer need this: reindex_all_load_state installs the persisted graphs itself and skips the build for every side it can supply. It is still used for blobs naming a rule this engine does not hold as approximate.

Called from restore_snapshot_state in db.rs after reindex.

pub fn hnsw_search_dst( &self, field: &str, dst_label: &str, q: &[f64], k: usize, ) -> Option<Vec<(u32, f64)>>

Find approximate nearest-neighbor ids on the dst side of the first approximate VectorSimilar rule covering (dst_label, field).

Returns None when no matching rule or HNSW index exists.

pub fn hnsw_search_dst_with_ef( &self, field: &str, dst_label: &str, q: &[f64], k: usize, ef: usize, ) -> Option<Vec<(u32, f64)>>

Self::hnsw_search_dst with an explicit layer-0 beam width, so a caller can widen the beam the same way exact VectorSimilar rules do.

pub fn hnsw_dst_len( &self, field: &str, dst_label: &str, q_len: usize, ) -> Option<usize>

Number of vectors in the dst-side index that would answer this query.

pub fn hnsw_has_rule(&self, field: &str) -> bool

Returns true if any approximate VectorSimilar rule covers field.

Use as a capability probe before calling hnsw_search_dst or hnsw_search_any_dst — presence of the rule guarantees the native Rust path will be used (HNSW when the index is populated, Rust brute-force otherwise); it does NOT guarantee a populated HNSW index.

pub fn hnsw_search_any_dst( &self, field: &str, q: &[f64], k: usize, ) -> Option<Vec<(u32, f64)>>

Like hnsw_search_dst but searches across all dst_labels that have an approximate VectorSimilar rule covering field.

Results from multiple rules are merged by node id (keeping the maximum score for any id that appears in more than one rule’s index), then sorted descending and truncated to k.

Returns None when no applicable rule has a populated HNSW index (same sentinel convention as hnsw_search_dst).

pub fn hnsw_search_any_dst_with_ef( &self, field: &str, q: &[f64], k: usize, ef: usize, ) -> Option<Vec<(u32, f64)>>

Self::hnsw_search_any_dst with an explicit layer-0 beam width.

pub fn hnsw_any_dst_len(&self, field: &str, q_len: usize) -> Option<usize>

Sum of vector counts across dst-side indexes covering field.

pub fn create_rule( &mut self, def: RuleDef, g: &mut GraphMut<'_>, ) -> Result<(), String>

Register a rule and backfill existing nodes. Returns Err on failed validate() or duplicate name.

pub fn delete_rule( &mut self, name: &str, g: &mut GraphMut<'_>, ) -> Result<(), String>

Remove the rule and exactly its owned edges. Returns Err if unknown.

pub fn on_node_changed( &mut self, n: u32, changed: Option<(&str, Option<Value>)>, g: &mut GraphMut<'_>, )

Called when node n is inserted (changed=None) or a field is updated.

• None: all rules where n’s label matches either side fire; index gains n.
• Some((field, old_value)): only rules watching field fire; index is updated using old_value for removal so stale buckets are cleaned.

For via-hop rules (def.via_label.is_some()), also fires when n carries the via-label: finds all srcs that route through n and recomputes their derived edges. Via-hop rules bypass the candidate index and use compute_desired_via instead.

Derived edges written here are chained into via-hop rules before the call returns (see RuleEngine::chain_from), so one write reaches a bounded fixpoint.

pub fn on_edge_changed( &mut self, etype_str: &str, src_id: u32, dst_id: u32, g: &mut GraphMut<'_>, )

Called when a user edge (etype_str, src_id, dst_id) is inserted or deleted (not a derived edge — those are managed by provenance, not here).

For any via-hop rule where via_edge == etype_str and src_id carries src_label, the src_id’s desired derived-edge set may have changed: a new WORKS_AT edge makes a new Org reachable as a via-node, and a deleted WORKS_AT removes a previously reachable Org.

This is the only hook the engine exposes for topology changes. It is called from db.rs on WalRecord::InsertEdge and WalRecord::DeleteEdge immediately after the topo is updated (so g.topo already reflects the new state), and re-entrantly by RuleEngine::chain_from for derived edges a rule just wrote.

pub fn on_node_removed(&mut self, n: u32, g: &mut GraphMut<'_>)

Retract every provenance edge touching n across all rules and drop n from every rule index using its current props.

Caller must invoke this while labels/props are still intact (before tombstone). Rules are walked in BTree name order; touching edges in BTree triple order. A second call on an already-retracted node is a no-op (crash-window replay / absent state).

Retractions chain: a retracted derived edge that some via-hop rule hops over retracts that rule’s edges too, bounded by MAX_CHAIN_DEPTH.

pub fn rebuild( &mut self, name: &str, g: &mut GraphMut<'_>, ) -> Result<(), String>

Recompute one rule from scratch. Only exit from the tripped latch.

If the full desired set fits in the budget, it is applied completely and tripped is cleared. If it still exceeds the budget, existing provenance is left completely untouched and tripped stays true (rebuild-is-noop for at/over-cap rules). Always counts as a fire evaluation per participating node. Returns Err if unknown.

A via-hop rule is never rebuilt through the candidate index — its predicate holds between the via node and the destination, which the index cannot express — so it goes through [apply_via_rebuild] or the via arm of [apply_streaming_rebuild_top_k] instead.

Trait Implementations

impl Debug for RuleEngine

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RuleEngine

fn default() -> RuleEngine

Returns the “default value” for a type.