Struct Db 

pub struct Db {
    pub objects: ObjectStore,
    pub id_index: IdIndex,
    pub del_index: IdIndex,
    pub sorted_indexes: SortedIndexes,
    pub graph: GraphStore,
    pub root: PathBuf,
    pub seq: AtomicU64,
    pub startup_ready: Arc<AtomicBool>,
    /* private fields */
}

Fields

objects: ObjectStore

id_index: IdIndex

del_index: IdIndex

Deleted id → its tombstone hash. The GRAVEYARD.

id_index answers “what is the current version of this key?”, so a delete has to remove the entry from it or the row would stay visible. But that made a deleted document’s whole HISTORY unreachable: AS OF enumerates ids from id_index, so the id was never considered at any sequence — even one long before the delete. Nothing was lost on disk (the tombstone node keeps a prev link to the full version chain); it was simply unreferenced.

That contradicted the central promise: a DELETE is a tombstone, not an erasure. So the pointer is not dropped, it is MOVED here — the id leaves the land of the living and stays addressable in history.

It is a second IdIndex rather than a new namespace inside the first because every operation needed — set, get, list, remove, WAL buffering, sharded on-disk layout — already exists and is already tested. A deliberately boring choice.

sorted_indexes: SortedIndexes

graph: GraphStore

root: PathBuf

seq: AtomicU64

startup_ready: Arc<AtomicBool>

True once startup is fully ready (MANIFEST loaded or cold scan complete). Warm starts set this true before returning from open(). Cold starts set this true in the background thread when scan completes. Writes are held with 503 until this is true; reads always proceed.

Implementations

impl Db

pub fn in_memory() -> Self

Create a pure in-memory database — no disk I/O, no migration, instant startup. Perfect for tests, hot-cache layers, and ephemeral sessions. All data is lost when the Db is dropped.

pub fn open(db_root: &Path, dek: Option<Dek>) -> Result<Self>

Open (or create) a database. Runs v1→v2 migration automatically if log.aof is present.

pub fn start_cold_scan(self_arc: Arc<Self>)

Call this from Manager::open_all() after Arc::new(db). Spawns the cold scan background thread with stable heap addresses. No-op if startup is already complete (warm start).

pub fn rebuild_id_index(&self) -> Result<usize>

Rebuild the id index from the object store, synchronously.

Every object carries its own coll, id and seq, so the id index is fully derivable: for each (coll, id) the highest seq wins. Use this to recover a database whose id-index WAL never reached disk — the objects are intact and verify, but list()/get() return nothing.

Idempotent, and safe on a healthy store (it rewrites the same winners). Returns the number of entries written. Flushes before returning.

pub fn repair(&self) -> Result<usize>

Full repair: rebuild the seq index and the id index from objects, even on a WARM store, then flush.

[start_cold_scan] deliberately no-ops when startup is already complete, which meant the documented repair path (“idempotent — a no-op on a warm store, a full self-heal on a stale MANIFEST”) could never repair a database that had a valid MANIFEST and a damaged id index. This is the forcing entry point; start_cold_scan keeps its O(1) warm-boot contract.

pub fn put( &self, coll: &str, id: &str, data: Value, caused_by: Vec<String>, valid_from: Option<String>, valid_to: Option<String>, ) -> Result<Node>

Write a document. Returns the new node with its content hash set.

Refuses an unusable or engine-owned collection name, and registers the collection if this is its first write — so that “this collection exists” becomes a durable fact at the moment it becomes true, rather than an inference drawn later from whatever the storage layer happens to have lying around.

pub fn collections(&self) -> Vec<String>

Every collection that currently exists.

THE authoritative answer, and the one a state root must commit to. Invariant across storage backends and independent of flush timing, because it reads recorded events rather than directory entries.

An empty-but-created collection is present here. That is the whole point: a database where orders was created and then emptied is not the same database as one where orders never existed, and a root that cannot tell them apart is not committing to the namespace.

pub fn collections_as_of(&self, target_seq: u64) -> Vec<String>

Which collections existed as of a sequence. The namespace is versioned for free, because the registry is ordinary documents in the DAG.

pub fn drop_collection(&self, coll: &str) -> Result<bool>

Drop a collection: record that the name is no longer live.

A TOMBSTONE, not an erasure — the same contract delete already has for documents. The registry keeps the name, marked dropped, so AS OF before the drop still reports the collection as having existed, and a later root can distinguish “dropped” from “never created”.

Documents are left where they are. Reclaiming them is compact’s job and an operator’s explicit decision; quietly destroying history behind a namespace operation is exactly the behaviour the engine refuses to have.

Returns false when the collection was not live to begin with.

pub fn state_root(&self) -> Result<StateRoot, String>

The database’s current state root.

A stateless recomputation over live state, not a maintained tree. That is a deliberate v1 choice: an incrementally-updated Merkle tree is a second source of truth that can silently drift from the first, and the cost of being wrong about a root is much higher than the cost of recomputing one.

pub fn state_root_as_of(&self, target_seq: u64) -> Result<StateRoot, String>

The state root as of a sequence.

Reuses the same enumeration AS OF queries already use — live ids plus the graveyard — so a historical root sees exactly what a historical query would see. Anything else would be a root for a state no query can return.

None when the material is gone: compact prunes superseded versions, and a root over history that has been discarded cannot be recomputed. Reported as unavailable rather than approximated.

pub fn create_root_at(&self, at_seq: u64) -> Result<RootRecord>

Persist the state root as of a sequence.

Creation and BACKFILL are the same operation with different arguments, and they are deliberately not the same COMMAND: at_seq at the tip is O(live state), while at_seq in the past is O(live state) plus a version-chain walk per document. Hiding the second behind something that looks like the first is how an operator discovers the cost by waiting.

pub fn create_root(&self) -> Result<RootRecord>

Persist the state root at the current tip.

pub fn get_root(&self, at_seq: u64) -> Option<RootRecord>

A persisted root record, if one was taken at this sequence.

pub fn list_roots(&self) -> Vec<RootRecord>

Every persisted root, oldest first.

pub fn verify_root(&self, at_seq: u64) -> RootVerification

Check a persisted root against a fresh recomputation.

Two INDEPENDENT facts, reported independently:

• the record exists and is well-formed
• the history needed to recompute it is still here

A persisted root may outlive the material that produced it — compact discards superseded versions, and after that a historical root is a perfectly valid record of something no longer reconstructable. Folding that into PASS would claim a verification that did not happen, and folding it into FAIL would report tampering that did not occur. So it is neither.

pub fn history_floor(&self) -> u64

The oldest sequence whose state can still be reconstructed.

0 until something prunes. compact records where it cut, because after it runs the engine cannot otherwise tell “this sequence had no writes” from “this sequence’s writes were discarded” — and those two answers differ by whether a failed verification means anything.

pub fn set_history_floor(&self, floor: u64) -> Result<()>

Declare where reconstructable history begins.

Public because pruning is not only something compact does: an operator who restores from a trimmed backup, or ships a database with its early segments removed, has pruned history that the engine has no way to notice. Without a way to say so, every historical root in that database would fail verification as if it had been tampered with.

MONOTONIC. The floor may rise and may never fall, because lowering it asserts that history exists which demonstrably does not — and the first thing that assertion does is turn an honest “unavailable” into a confident, wrong “mismatch”.

pub fn put_batch( &self, ops: Vec<(String, String, Value, Vec<String>, Option<String>, Option<String>)>, ) -> Result<Vec<Node>>

Batch put: write N documents in parallel, preserving monotonic seq ordering. Pre-allocates N seq numbers atomically, then parallelises object writes and id-index updates via Rayon. Each op is independent — safe to parallelise. Returns nodes in input order with assigned seq numbers.

pub fn try_flush_all(&self) -> Result<()>

Flush both the id-index WAL and MANIFEST, REPORTING failure.

This is the durability boundary: until it returns Ok(()), writes that put() acknowledged may not be on disk. Callers that must not lose data — anything about to take a destructive or externally-visible action on the strength of a persisted record — should use this, not [flush_all].

Every stage is attempted even if an earlier one fails (a MANIFEST flush is still worth doing when one index leaf failed), and the first error is returned. Failed id-index entries stay in the WAL for retry.

pub fn flush_all(&self)

Flush both the id-index WAL and MANIFEST. Used on graceful shutdown.

Errors are logged, not returned — kept for back-compat and for the ticker/Drop paths that have nowhere to propagate. Prefer [try_flush_all] whenever the outcome matters.

pub fn compact(&self) -> Result<CompactStats>

Compact the v3 packed object store: keep the CURRENT version of every document (from the id-index) and reclaim everything else. No-op unless running with the v3 segment substrate (--dag-v3 / NEDB_DAG_V3).

This is a PRUNING operation: superseded/historical object versions are dropped, so AS OF / TRACE over pruned versions is discarded — that is what reclaims the space. Flushes first so all data is durable on disk before the old segments are deleted. Reclaim space by rewriting the segments with only CURRENT versions.

This discards history. On purpose.

The live set is each document’s current-version hash and nothing else, so compaction drops every superseded version and every tombstone. After it runs, AS OF can no longer reach a prior value and TRACE can no longer walk to a pruned ancestor — the rows simply become unavailable rather than wrong, and verify() stays clean because what remains is still internally consistent.

That is worth stating loudly, because NEDB’s headline property is that history is permanent and never garbage-collected — and it is, right up until an operator calls THIS. Nothing calls it automatically: it is not on the HTTP surface, not in the CLI, and not on any timer. It exists for the operator who has decided, explicitly, to trade the audit trail for disk space.

A graveyard entry whose tombstone was pruned is left pointing at an object that no longer exists. get_as_of degrades to None there rather than failing, so a compacted store answers “not available at that sequence” instead of erroring or inventing a value.

Live branches veto it

A branch promises a future three-way merge, and a three-way merge needs the BASE side: the parent state as of the branch’s fork point. This prunes every superseded version down to the tip, which is exactly the material that base is made of. Running it under a live branch would produce “branch exists, merge ancestry gone” — a branch that can never be reconciled and does not find that out until someone tries.

Because compaction here is all-or-nothing to the tip, there is no honest partial answer (“prune down to the pin” is a different algorithm, not a parameter). So the answer is REFUSAL, naming the branches and what they pin. There is deliberately no force flag: a bypass would be reached for exactly when it does the damage, and a silent bypass is the thing this interlock exists to design out. The operator’s escape hatch is to merge or abandon the branch — both of which are recorded decisions.

pub fn flush_manifest_if_dirty(&self)

Flush MANIFEST to disk if dirty. No-op for in-memory databases.

pub fn try_flush_manifest(&self) -> Result<()>

Atomically persist current seq+head to MANIFEST, reporting failure. No-op (Ok) for in-memory databases.

A silently failed MANIFEST write is not data loss — the startup self-heal rescans — but it IS a warm-boot regression and, on a full disk, the first symptom that persistence is failing. Callers deserve to know.

pub fn flush_manifest(&self)

Atomically persist current seq+head to MANIFEST. No-op for in-memory databases. Errors are logged; prefer [try_flush_manifest] when the outcome matters.

pub fn embedded_flush_interval_ms() -> Option<u64>

Start a background thread that flushes both the id-index WAL and MANIFEST every interval_ms milliseconds. Call this after Arc::new(db) — the Arc keeps Db alive for the thread’s lifetime. Flush cadence for EMBEDDED durable handles (the napi and pyo3 open() paths).

nedbd has always run the manifest ticker at 1 s, so a server flushes the id-index WAL and MANIFEST every second and a hard kill loses at most a second of acknowledged writes. The embedded bindings did not start a ticker at all: their WAL was flushed only by the exit hooks (SIGINT/SIGTERM/atexit) — so an embedded app killed with SIGKILL, OOM-killed, or cut by power lost EVERY write since open, with no bound. Found by CHALK / Sports-Rater on 2026-09-04 (acknowledged fan writes gone after kill -9). Since 2.8.5 the bindings start the ticker on durable open with this cadence — parity with nedbd.

NEDB_FLUSH_MS overrides: an integer of milliseconds (min 50), or 0 / off to disable (only for hosts that own their own flush cadence). Unset → 1000.

pub fn start_manifest_ticker(self_arc: Arc<Self>, interval_ms: u64)

Spawn the background flush ticker.

The ticker holds a Weak<Db> and exits the first time the upgrade fails — i.e. as soon as the last real owner drops the database. The caller must therefore keep its own Arc alive for as long as it wants ticking; every current caller already does (nedbd stores it in its database map, the napi and pyo3 handles own theirs).

It used to hold a strong Arc inside an unconditional loop, which meant the thread never exited and the Db was never dropped. Three consequences, all of them live since 2.8.5:

• The exclusive data-dir LOCK taken in Db::open was never released, so reopening the same path in the same process failed with “locked by another process (pid N)” where N was the caller’s own pid.
• Every open() leaked a thread and the entire Db — indexes, caches, segment handles — for the lifetime of the process.
• Drop for Db (flush-on-close) could never fire for embedded users, exactly as its own doc comment warned: it “only fires once every owning handle is gone”, and an immortal thread always held one.

nedbd’s drop_db was hit by the same thing: removing a database from the map did not free it, and an orphaned ticker went on fsyncing it.

The Arc is upgraded inside the loop and dropped before the next sleep, so the ticker never extends the database’s life across a tick. No final flush is needed here — the owner’s Drop does it.

pub fn head(&self) -> String

Return the current Merkle head string. O(1) — read from cache.

pub fn delete(&self, coll: &str, id: &str) -> Result<bool>

Delete a document — writes a tombstone node and removes the id from the index. The object history is preserved in the DAG; only the live id pointer is cleared.

pub fn get(&self, coll: &str, id: &str) -> Option<Node>

Get the current version of a document by id.

pub fn get_by_hash(&self, hash: &str) -> Option<Node>

Get a specific version of a document by object hash.

pub fn get_as_of(&self, coll: &str, id: &str, target_seq: u64) -> Option<Node>

Get a document AS OF a specific sequence number. Walks the version chain (prev links) backward until seq <= target.

Reaches DELETED documents too. A delete moves the id’s pointer into the graveyard rather than dropping it, so the version chain stays walkable and a row is still readable at a sequence before it was deleted — which is what “a DELETE is a tombstone, not an erasure” has to mean in practice. At or after the tombstone’s own sequence the document is correctly absent.

pub fn list_ids_including_deleted(&self, coll: &str) -> Vec<String>

Every id in a collection that AS OF must consider: the live ones, plus the deleted ones whose history is still addressable.

Order is stable (sorted, deduplicated) so a historical query answers the same way run to run.

pub fn list(&self, coll: &str) -> Vec<Node>

List all documents in a collection, returning current versions.

pub fn range_scan( &self, coll: &str, field: &str, low: Option<&Value>, high: Option<&Value>, low_incl: bool, high_incl: bool, ) -> Option<Vec<Node>>

Candidate nodes whose field falls in the given range, via the sorted index. None when no index covers (coll, field) — the caller must then fall back to a scan.

Returns CURRENT versions only (the index drops a superseded hash on overwrite), so this must not be used to serve an AS OF query.

pub fn index_lookup( &self, coll: &str, field: &str, values: &[Value], ) -> Option<Vec<Node>>

Candidate nodes whose field equals any of values — the indexed path for = and for IN (...). None when no index covers the field.

pub fn range_cardinality( &self, coll: &str, field: &str, low: Option<&Value>, high: Option<&Value>, low_incl: bool, high_incl: bool, ) -> Option<usize>

How many rows an indexed range covers, without reading any of them. None when no index covers the field.

pub fn has_sorted_index(&self, coll: &str, field: &str) -> bool

True when a sorted index covers (coll, field).

pub fn order_by_asc(&self, coll: &str, field: &str, limit: usize) -> Vec<Node>

ORDER BY field ASC LIMIT n — uses sorted index if available, else falls back to full scan.

pub fn order_by_desc(&self, coll: &str, field: &str, limit: usize) -> Vec<Node>

ORDER BY field DESC LIMIT n

pub fn trace(&self, hash: &str, reverse: bool, limit: usize) -> Vec<Node>

TRACE caused_by — walk causal graph from a node.

pub fn verify(&self) -> (usize, Vec<String>)

Verify tamper-evidence of all objects.

pub fn create_sorted_index(&self, coll: &str, field: &str)

Create a sorted index for a (coll, field) pair.

pub fn get_hash_by_seq(&self, seq: u64) -> Option<String>

Resolve a sequence number to its content hash (v1 compatibility). Only covers nodes written in the current process session + cold-scan nodes.

pub fn tip(&self) -> Option<Node>

The tip — the most recently written node (highest seq), or None if the database is empty. O(1): self.seq is the next-to-assign counter, so the latest write sits at seq - 1; we resolve it through the same seq_index → object-store path a normal read uses, so the returned Node is byte-identical to one fetched by id or hash (it carries its own seq, hash, causal links, and valid-time). This is the cheap “give me the latest write” primitive — the head of the log, not an aggregate.

pub fn tip_collection(&self, coll: &str) -> Option<Node>

The collection-local tip — the most recent write into coll (highest seq in that collection), or None if the collection has no writes. O(1): resolves through coll_tip_hash, a dedicated per-collection map kept current on every write (update_head), restored from MANIFEST on warm boot, and rebuilt by the cold scan — durable across restarts by construction, same contract as tip() for the global head. Conceptually a different index than the global tip() (global head vs collection head), kept as a separate method so each is explicit — parity with the Python reference’s tip(coll). Lets a consumer resume one chain (e.g. blocks / tx / utxo) without pulling global tip and filtering.

pub fn since(&self, after_seq: u64, limit: usize) -> SinceBatch

Changefeed page: up to limit nodes written AFTER after_seq (EXCLUSIVE), ascending by seq, wrapped in a SinceBatch cursor envelope. after_seq is the cursor you last applied (a prior tip() seq or to_seq). limit bounds the page — 0 means DEFAULT_SINCE_LIMIT, so the engine primitive can never materialize an unbounded batch even when embedders call it directly (the safety is here, not only in the HTTP layer). Drain by paging while has_more, advancing your cursor to to_seq, then hand off to the live subscribe edge. The append-only log IS the changefeed, so this is an O(page) walk; unresolved seqs (outside seq_index coverage — see scan_status()) are skipped rather than faked.

pub fn scan_status(&self) -> ScanStatus

Replication readiness — see ScanStatus. scan_complete gates safe historical catch-up: a consumer pulling an old cursor right after a cold start must wait for it, or since() may hand back a partial page that looks like “caught up”. Computes the indexed range by scanning the in-memory seq index (O(index)) — intended for periodic status polls, not the per-write hot path.

pub fn link(&self, frm: &str, rel: &str, to: &str) -> Result<()>

Add an explicit named relation edge between two documents. Add an explicit named relation between two “coll:id” nodes. Relations stored as links documents — NQL-queryable, time-travelable, consistent with the PyO3 binding which uses the same links convention.

pub fn unlink(&self, frm: &str, rel: &str, to: &str) -> Result<bool>

Remove a named relation (deletes the links document).

pub fn neighbors(&self, frm: &str, rel: &str) -> Vec<Node>

Get neighbor nodes via a named relation. Queries links — consistent with the PyO3 binding.

impl Db

pub fn install_exit_flush(self_arc: Arc<Db>)

Flush this durable database’s buffered state on SIGINT/SIGTERM (Ctrl+C, kill, orchestrator shutdown) — the flush-on-close contract extended to hard exits that never run Drop.

Call once, after the database is wrapped in an Arc (the registry holds a Weak, so this never keeps the Db alive). Idempotent; safe to call from multiple databases. A no-op for in-memory (:memory:) databases.

let db = Arc::new(Db::open(std::path::Path::new("/data/mydb"), None)?);
Db::install_exit_flush(Arc::clone(&db));   // durable across Ctrl+C / SIGTERM

Trait Implementations

impl Drop for Db

fn drop(&mut self)

Flush buffered state when the database is closed so a write-then-drop sequence is durable without an explicit flush_all().

IdIndex::set only stages updates in the in-memory WAL write_buf; disk persistence happens in flush_write_buf(), normally driven by the manifest ticker. A short-lived Db (a library user’s { let db = Db::open(p)?; db.put(..)?; } block, or a test) has no ticker, so without this its writes would be silently lost on reopen. Flushing on drop mirrors the flush-on-close contract of other embedded stores (sled, RocksDB).

In production this is a harmless safety net, not the primary durability path: the manifest ticker thread holds an Arc<Db> for the process lifetime, so Drop only fires once every owning handle is gone. No-op for in-memory databases (flush_all short-circuits on :memory:).

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.