nodedb_lite/engine/vector/durable/mod.rs
1// SPDX-License-Identifier: Apache-2.0
2
3//! Durable per-document vector storage — the source of truth for vectors.
4//!
5//! # Why this exists
6//!
7//! A vector used to live in exactly one place: the in-memory HNSW index, which
8//! reached disk only when `flush` wrote the `vec/hnsw/<collection>` segment.
9//! That gave vectors a weaker durability guarantee than the documents they
10//! belong to — a document is durable the moment its write is acknowledged
11//! (versioned put), while its vector survived only if a later flush happened to
12//! run. An unclean exit therefore lost every vector written since the last
13//! flush, silently, because the write had already reported success.
14//!
15//! It also made the segment the *only* copy, so a segment that could not be
16//! reopened left exactly two options, both wrong: keep a checkpoint whose node
17//! vectors are empty placeholders (the first distance computation panics with
18//! `dist_to_node: byte-length mismatch`), or drop the index and lose every
19//! vector permanently. There was no third option because nothing else held the
20//! data — in particular the CRDT holds only `embedding_dim`, never the floats,
21//! so the "rebuild from CRDT" the restore path spoke of could never have worked.
22//!
23//! # The contract
24//!
25//! Every vector is written here in the same operation that makes its document
26//! durable. [`crate::engine::vector::pagedb_backing`] segments become a
27//! *derived* index: an accelerator that can always be rebuilt from these rows
28//! ([`load_collection`]), never the master copy. That makes a corrupt or
29//! unreadable segment a rebuild, not a data-loss event.
30//!
31//! # Layout
32//!
33//! `Namespace::Vector`, one row per `(index key, document id)`; see
34//! [`layout`] for the key. The value is little-endian `f32` values with no
35//! header. The dimension is implied by the byte length, which is why
36//! `decode` rejects a length that is not a multiple of 4. Databases written
37//! in the earlier key layout are rewritten once on open by [`migrate`].
38
39mod layout;
40pub(crate) mod migrate;
41mod rows;
42
43#[cfg(not(target_arch = "wasm32"))]
44pub(crate) use rows::rebuild_index;
45pub(crate) use rows::{list_collections, load_collection, put_op, remove};