Expand description
Incremental adjacency: per-commit delta segments (#765).
Each pure-append topology commit writes one tiny Parquet file
indexes/adjacency/deltas/<generation>.parquet holding exactly the edges it
created (possibly zero rows, for a node-only flush). The adjacency provider
serves a fresh view as base CSR ⊎ a contiguous chain of segments covering
(G_base, G_cur], so newly-created edges are visible without a full rebuild.
Anything that breaks the chain — a DELETE, a crash between commit and segment
write, an unreadable segment, or a chain longer than MAX_DELTA_CHAIN —
reads as stale and falls back to the existing full-rebuild path.
The merge (apply_delta_segments) reconstructs the base entries from the
loaded CSR, concatenates the (filtered) delta entries, and re-runs the
builder’s csr_from_entries. Because
edge_id is a unique surrogate, the result is byte-identical to a full
rebuild from topology/ for the same edges — that equivalence is the
correctness contract (acceptance criterion 1), pinned by the tests below.
Structs§
- Delta
Edge - One created edge in a delta segment, in creation (ascending
edge_id) order. - Delta
Segment - The edges created by the commit that bumped the counter to
generation.
Constants§
- MAX_
DELTA_ CHAIN - Longest delta chain served before the index reads stale and is rebuilt.
Bounds both per-view merge cost and
deltas/accumulation between rebuilds.
Functions§
- apply_
delta_ segments - Merge a contiguous delta
chainontobase(a loaded CSR fordirection) and return the overlaid CSR — equal to a full rebuild over the base’s edges plus the chain’s, forstem. - delta_
dir indexes/adjacency/deltas/withinproject_dir.- delta_
path - Path of the delta segment for
generation. - discard_
segment - Best-effort removal of the segment at
generation— for a commit that bumps the counter but writes no segment (any DELETE / non-pure-append statement). This makes “no segment here” an affirmative invariant: the chain can never contain a file the bumping commit did not author, even across a counter reset that left a stale file at that generation. A failure is harmless — an unexpected file atgenerationonly breaks the chain there, forcing a (correct) rebuild. - prune_
delta_ segments - Remove every delta segment with generation
<= up_to(consumed by a rebuild or compaction atup_to). Best-effort: a failed unlink leaves dead weight a later prune removes, never incorrect data. Segments> up_to(written by a concurrent append during the build) survive, so the new base + those is immediately fresh. - read_
delta_ chain - Read the contiguous chain of segments covering
(base, current]. - read_
delta_ segment - Read one delta segment from disk. A missing file is an error here — callers
that tolerate absence (
read_delta_chain) check existence first. - write_
delta_ segment - Write the segment for
generation(creatingdeltas/if needed). A zero-edge segment is still written so a node-only commit keeps the chain contiguous. Atomic via temp-file + rename (RewriteBatch).