Skip to main content

Module segment

Module segment 

Source
Expand description

segment.rs — NEDB v3 packed object substrate.

v3 keeps the v2 logical model intact (content-addressed, immutable, BLAKE2b-verified DAG nodes) and changes only where the bytes live: instead of one filesystem object per node — which caps throughput at the OS small-file metadata rate — many immutable objects are appended into segment files addressed through an in-memory hash -> (segment, offset, len) index. The hash is still BLAKE2b(content) and is re-verified on every read, so content-addressing and tamper-evidence are unchanged.

This module knows nothing about Node, JSON, or encryption: callers (ObjectStore) pass already-serialized/encrypted content bytes and the precomputed hash. That keeps the segment store a pure content<->location layer and leaves all crypto/serialization in store.rs.

Phases:

  1. Segment append + in-memory index + startup scan + tail-truncation.
  2. Compaction/pruning — rewrite the live object set into fresh segments, reclaiming dead (superseded/spent) records. compact(live).
  3. On-disk .idx sidecars — a sealed segment gets a checksummed hash -> (offset,len) index file so cold start loads it instead of rescanning every record. Missing/corrupt .idx falls back to a scan.

Opt-in only: ObjectStore instantiates this when NEDB_DAG_V3 is set (surfaced as the --dag-v3 flag). Default storage is byte-for-byte v2.

Structs§

CompactStats
Result of a compact() pass.
SegmentStore
Append-only, content-addressed packed object store with an in-memory index.