Struct RetrievalEngine 

pub struct RetrievalEngine { /* private fields */ }

Multi-modal retrieval engine with production vector search

Lock Ordering (SHO-72)

To prevent deadlocks, locks MUST be acquired in this order:

1. vector_index - Vector similarity search index
2. id_mapping - Memory ID ↔ Vector ID mapping
3. consolidation_events - Introspection event buffer

Rules:

• Never acquire a higher-numbered lock while holding a lower-numbered lock
• For read operations, prefer read() over write() when possible
• Release locks as soon as possible (don’t hold during I/O)

Note: Memory graph (Hebbian learning) has been consolidated into GraphMemory which is managed at the API layer (MultiUserMemoryManager.graph_memories)

Implementations

impl RetrievalEngine

pub fn new( storage: Arc<MemoryStorage>, embedder: Arc<MiniLMEmbedder>, ) -> Result<Self>

Create new retrieval engine with shared embedder (CRITICAL: embedder loaded only once)

ATOMIC ARCHITECTURE: RocksDB is the ONLY source of truth.

• Vector mappings are stored atomically with memories in RocksDB
• Vamana index is rebuilt from RocksDB on startup (pure in-memory cache)
• No more file-based IdMapping = no more orphaned memories

pub fn with_event_buffer( storage: Arc<MemoryStorage>, embedder: Arc<MiniLMEmbedder>, consolidation_events: Option<Arc<RwLock<ConsolidationEventBuffer>>>, ) -> Result<Self>

Create retrieval engine with event buffer for consolidation introspection

The event buffer is used to record Hebbian learning events:

• Edge formation (new associations)
• Edge strengthening (co-activation)
• Edge potentiation (LTP)
• Edge pruning (decay below threshold)

ATOMIC STARTUP: Rebuilds Vamana from RocksDB mappings for crash safety.

pub fn set_consolidation_events( &mut self, events: Arc<RwLock<ConsolidationEventBuffer>>, )

Set the consolidation event buffer (for late binding after construction)

pub fn save(&self) -> Result<()>

Save Vamana index to disk for instant startup

HYBRID ARCHITECTURE:

• RocksDB: Source of truth for memories and ID mappings
• .vamana file: Persisted graph for instant startup (skip rebuild)

On next startup, if .vamana exists and is valid, we load it directly. Otherwise, we fall back to rebuilding from RocksDB.

pub fn len(&self) -> usize

Get number of vectors in the index

pub fn is_empty(&self) -> bool

Check if index is empty

pub fn get_indexed_memory_ids(&self) -> HashSet<MemoryId>

Get set of all indexed memory IDs (for integrity checking)

pub fn index_memory(&self, memory: &Memory) -> Result<()>

Add memory to vector index with atomic RocksDB storage

ATOMIC ARCHITECTURE: This method stores the vector mapping atomically in RocksDB alongside the memory data, ensuring no orphaned memories.

For long content, this chunks the text and creates multiple embeddings to ensure ALL content is searchable, not just the first 256 tokens.

pub fn reindex_memory(&self, memory: &Memory) -> Result<()>

Re-index an existing memory with updated embeddings

Used when memory content is updated via upsert() to ensure the vector index reflects the new content.

Strategy: Remove old vector and add new one (Vamana doesn’t support update-in-place)

pub fn remove_memory(&self, memory_id: &MemoryId) -> bool

Remove a memory from the vector index

ATOMIC ARCHITECTURE: Removes the vector mapping from RocksDB atomically. The in-memory Vamana index is updated immediately, and the RocksDB mapping is deleted to ensure consistency on restart.

Returns true if the memory was found and removed, false if not indexed.

pub fn search_ids( &self, query: &Query, limit: usize, ) -> Result<Vec<(MemoryId, f32)>>

Search for memory IDs only (for cache-aware retrieval)

With chunked embeddings, multiple vectors can map to the same memory. This function deduplicates by MemoryId, keeping the highest-scoring chunk.

Returns (MemoryId, similarity_score) pairs

pub fn get_from_storage(&self, id: &MemoryId) -> Result<Memory>

Get memory from storage by ID

pub fn search_by_embedding( &self, embedding: &[f32], limit: usize, exclude_id: Option<&MemoryId>, ) -> Result<Vec<(MemoryId, f32)>>

Search for similar memories by embedding directly (SHO-106)

Used for interference detection to find memories similar to a new memory. Optionally excludes a specific memory ID from results.

With chunked embeddings, multiple vectors can map to the same memory. This function deduplicates by MemoryId, keeping the highest-scoring chunk.

Returns (MemoryId, similarity_score) pairs

pub fn search(&self, query: &Query, limit: usize) -> Result<Vec<SharedMemory>>

Search for memories using multiple retrieval modes (zero-copy with Arc)

pub fn matches_filters(&self, memory: &Memory, query: &Query) -> bool

Check if memory matches query filters

Delegates to Query::matches() which is the SINGLE source of truth for all filter logic. This ensures consistent filtering across all memory tiers and retrieval modes.

pub fn rebuild_index(&self) -> Result<()>

Build vector index from existing memories (resumable on failure)

Uses incremental indexing so partial progress is preserved:

• Skips memories already in the index
• On failure, next rebuild/repair continues from where it left off
• Logs progress every 1000 memories for monitoring

pub fn add_to_graph(&self, _memory: &Memory)

👎Deprecated:

Use GraphMemory at API layer instead

Add memory to knowledge graph - DEPRECATED Use GraphMemory at the API layer instead

pub fn record_coactivation(&self, _memory_ids: &[MemoryId])

👎Deprecated:

Use GraphMemory.record_memory_coactivation() at API layer instead

Record co-activation of memories - DEPRECATED Use GraphMemory.record_memory_coactivation() at API layer instead

pub fn graph_maintenance(&self)

👎Deprecated:

Use GraphMemory.apply_decay() at API layer instead

Perform graph maintenance - DEPRECATED Use GraphMemory.apply_decay() at API layer instead

pub fn graph_stats(&self) -> MemoryGraphStats

👎Deprecated:

Use GraphMemory.get_stats() at API layer instead

Get memory graph statistics - DEPRECATED Use GraphMemory.get_stats() at API layer instead

pub fn auto_rebuild_index_if_needed(&self) -> Result<bool>

Check if vector index needs rebuild and rebuild if necessary

Returns true if rebuild was performed.

IMPORTANT: We perform a full rebuild from RocksDB rather than using Vamana’s internal auto_rebuild_if_needed(). The internal rebuild extracts live vectors and assigns new sequential IDs (0, 1, 2, …), but does NOT update the RetrievalEngine’s id_mapping. This would silently corrupt all search results after compaction — searches would return wrong memories because old vector_id→memory_id mappings no longer match the new vector IDs.

By rebuilding from RocksDB (the single source of truth), both the vector index and the id_mapping are rebuilt atomically.

pub fn index_health(&self) -> IndexHealth

Get vector index degradation info

impl RetrievalEngine

pub fn search_tracked( &self, query: &Query, limit: usize, ) -> Result<TrackedRetrieval>

Search with tracking for later feedback

Use this when you want to provide feedback on retrieval quality. Returns a TrackedRetrieval that can be used with reinforce_recall.

pub fn reinforce_recall( &self, memory_ids: &[MemoryId], outcome: RetrievalOutcome, ) -> Result<ReinforcementStats>

Reinforce memories based on task outcome (core feedback loop)

This is THE key method that closes the Hebbian loop:

• If outcome is Helpful: strengthen associations, boost importance
• If outcome is Misleading: weaken associations, reduce importance
• If outcome is Neutral: just record access (mild reinforcement)

Call this after a task completes to indicate which memories helped.

pub fn reinforce_tracked( &self, tracked: &TrackedRetrieval, outcome: RetrievalOutcome, ) -> Result<ReinforcementStats>

Reinforce using a tracked retrieval (convenience wrapper)

pub fn reinforce_batch( &self, feedbacks: &[RetrievalFeedback], retrieval_memories: &HashMap<String, Vec<MemoryId>>, ) -> Result<Vec<ReinforcementStats>>

Batch reinforce multiple retrievals (for async feedback processing)