Struct StorageEngine 

pub struct StorageEngine { /* private fields */ }

The unified storage engine for MenteDB.

Coordinates page allocation, caching, and write-ahead logging to provide crash-safe, page-oriented storage for memory nodes.

Concurrency model (inspired by WAL-mode databases):

• Reads are lock-free: read_page only touches the buffer pool and page manager — no file locks, no WAL access.
• Writes are fully serialized via a blocking flock(2) on the WAL file. The entire write transaction (page allocation + WAL append + page write + fsync) executes under a single flock, ensuring correctness across multiple processes sharing the same data directory.
• State is refreshed from disk under the flock: page count is re-read from the file header and LSN is re-read from the WAL tail, so no process can act on stale in-memory state.
• One process per database. Open acquires an exclusive lock on a LOCK file in the data directory and holds it for the engine’s lifetime. The per operation flock protects individual write transactions, but layers above the storage engine cache state across operations (buffer pool, page map, index snapshots), and a second concurrent process flushing those caches rolls the database back to its own open time snapshot. A second open therefore fails fast with a “locked” error instead of silently corrupting state; callers retry until the owner exits.

Implementations

impl StorageEngine

pub fn open(path: &Path) -> MenteResult<Self>

Open (or create) a storage engine rooted at path.

path must be a directory; it will be created if it does not exist. After opening, any uncommitted WAL entries are replayed for crash recovery.

Example

use mentedb_storage::StorageEngine;

let engine = StorageEngine::open("/tmp/mentedb".as_ref())?;
// engine is ready — WAL recovery already ran if needed

pub fn recover(&self) -> MenteResult<usize>

Replay WAL entries to recover writes that were not checkpointed.

For each PageWrite entry the serialized data is written back to its page. After replay the WAL is truncated. Returns the number of entries replayed.

pub fn close(&self) -> MenteResult<()>

Gracefully shut down: flush dirty pages, sync files.

Example

engine.close()?;

pub fn allocate_page(&self) -> MenteResult<PageId>

Allocate a fresh page (for internal/test use).

WARNING: In multi-process scenarios, prefer store_memory which allocates under the WAL flock. This method does NOT acquire the flock.

pub fn read_page(&self, page_id: PageId) -> MenteResult<Box<Page>>

Read a page through the buffer pool (lock-free — no WAL access).

pub fn write_page(&self, page_id: PageId, data: &[u8]) -> MenteResult<()>

Write data into an already-allocated page with WAL protection.

Acquires the WAL flock for the duration of the write transaction. For new pages, prefer store_memory which allocates + writes atomically.

pub fn store_memory(&self, node: &MemoryNode) -> MenteResult<PageId>

Serialize and store a MemoryNode into a single page.

The entire operation — page allocation, WAL append, page write — executes under a single WAL flock, making it safe across multiple processes.

Example

use mentedb_storage::StorageEngine;
use mentedb_core::{MemoryNode, memory::MemoryType, types::AgentId};

let engine = StorageEngine::open("/tmp/mentedb".as_ref())?;
let node = MemoryNode::new(
    AgentId::new(),
    MemoryType::Semantic,
    "User likes dark mode".to_string(),
    vec![0.1, 0.2],
);
let page_id = engine.store_memory(&node)?;

pub fn store_memory_batch( &self, nodes: &[MemoryNode], ) -> MenteResult<Vec<PageId>>

Store multiple MemoryNodes in a single locked transaction.

Acquires the WAL flock once, writes all nodes, then releases. This avoids the per-write overhead of reload_header / reload_lsn for bulk inserts. Auto-checkpoints after the batch if the WAL exceeds the threshold.

pub fn update_memory( &self, page_id: PageId, node: &MemoryNode, ) -> MenteResult<()>

Update a MemoryNode in place on its existing page.

The write goes through the WAL-protected write_page path, so it is crash-durable and keeps the buffer pool coherent. Unlike storing to a fresh page, this never orphans the old copy.

pub fn delete_memory(&self, page_id: PageId) -> MenteResult<()>

Delete a memory by returning its page to the free list.

The deletion is WAL-logged before the page is freed, so it survives a crash: recovery replays the PageFree entry and the memory does not resurrect on reopen. The freed page is reused by later allocations.

pub fn load_memory(&self, page_id: PageId) -> MenteResult<MemoryNode>

Load and deserialize a MemoryNode from the given page.

Example

let node = engine.load_memory(PageId(1))?;
println!("memory: {}", node.content);

pub fn checkpoint(&self) -> MenteResult<()>

Checkpoint: flush all dirty pages, sync to disk, and truncate the WAL.

Example

// After a batch of writes, checkpoint to reclaim WAL space
engine.checkpoint()?;

pub fn scan_all_memories(&self) -> Vec<(MemoryId, PageId)>

Scan all pages and return (MemoryId, PageId) pairs for every valid memory node.

Refreshes the page count from disk before scanning so pages written by other processes are included. Used to rebuild the page map on startup.

Example

let memories = engine.scan_all_memories();
for (memory_id, page_id) in &memories {
    println!("{memory_id} -> page {}", page_id.0);
}