Crate fathomdb_engine

Expand description

FathomDB engine — the runtime core: storage, projections, ingest and query.

⚠ Most consumers should depend on the fathomdb facade crate instead. fathomdb re-exports exactly the governed application surface and gates the operator/recovery seam behind a cargo feature; this crate is the implementation and exposes internals the facade deliberately withholds. Depend on it directly only if you are building FathomDB tooling.

§What it does

One Engine owns an embedded SQLite database (FTS5 + sqlite-vec), the single writer thread, a thread-affine reader pool serving DEFERRED-tx snapshots, the background projection scheduler, and — optionally — an in-process embedder. There is no server and no sidecar.

Hybrid retrieval. A vector branch and an FTS5 branch, fused by Reciprocal Rank Fusion on ordinal rank (never on raw, non-comparable scores), with an optional CPU cross-encoder rerank and an optional graph-BFS third arm over temporal fact edges.
Canonical rows + projections. Writes land as durable canonical rows; FTS, vector and attribute indexes are engine-maintained projections rebuildable from them.
Record lifecycle. Transaction-time supersession keyed on logical_id, an existence axis (transition / purge), and world-time validity windows on nodes plus t_valid / t_invalid on edges.
Deletion on request. Engine::erase_source erases every row carrying a provenance id — including anonymous rows Engine::purge cannot reach — and finishes the erasure at rest.

§Provenance is mandatory

PreparedWrite::Node and PreparedWrite::Edge carry source_id: SourceId, a newtype rather than an Option<String>. Engine::erase_source addresses rows by source_id, so a row written without one could never be erased; SourceId::new is the only public constructor and makes that state inexpressible.

§Stability

Pre-1.0, so beta. SCHEMA_VERSION is the on-disk contract; migrations run at open and only there. PreparedWrite, SearchFilter and EngineError are #[non_exhaustive] or documented as additive.

Re-exports§

pub use lifecycle::Subscription;

Modules§

lifecycle: Lifecycle observability data types.

Structs§

Bm25fFieldWeights: F5 (0.8.14 Slice 10) — per-field BM25F weights for the search_index_v2 multi-column FTS index. One weight per indexed field (kind/body/status), applied as the field’s contribution multiplier in the BM25F weighted-term-frequency accumulation.
Bm25fQueryPlan: F5 (0.8.14 Slice 10) — the compiled BM25F query plan for the fielded lexical arm (ADR-0.8.1 §3.2 BM25fQueryPlan). Carries the tunable per-field weights and the tunable length-normalization b (and the term-saturation k1).
BoundaryCrossing: 0.8.20 Slice 10b (R-20-NV) — one node that crossed a validity boundary inside the interrogated interval, as reported by Engine::crossed_boundary_since.
CheckIntegrityOpts: Doctor check-integrity invocation flags. quick and round_trip are accepted in 0.6.0 but treated as default; only full activates PRAGMA integrity_check. Per dev/design/recovery.md § Doctor-only flags.
ConsolidateAxis: 0.8.12 Slice 15 (OPP-2, ADR-0.8.12) — one (subject-entity, relation) axis to consolidate via Engine::consolidate_with_provider. FathomDB assembles the competing fact-edge cluster for this axis DETERMINISTICALLY (CPU-only, no LLM) by querying active canonical_edges where from_id = subject_logical_id AND kind = relation.
ConsolidateCandidateEdge: 0.8.12 Slice 15 (OPP-2, ADR-0.8.12) — one competing fact-edge in a candidate cluster sent to the consolidation harness. Assembled deterministically from canonical_edges; sent to the harness as the request payload; the harness’s verdict references edges back by edge_ref (the edge’s stable logical_id).
ConsolidateReceipt: 0.8.12 Slice 15 (OPP-2, ADR-0.8.12) — receipt returned by Engine::consolidate_with_provider. Consolidation records supersession / recency METADATA only (§2.1): edge bodies are never rewritten and no row is ever deleted, so these counts describe metadata transitions, not content changes.
CorruptionDetail: Stable corruption-on-open detail carried by EngineOpenError::Corruption.
CounterSnapshot: Snapshot of engine-internal counters returned by Engine::counters.
DumpProfileReport: Result of [Engine::dump_profile]. Mirrors the open-time embedder posture + the per-kind vector configuration registered in _fathomdb_vector_kinds.
DumpRowCountsReport: Result of [Engine::dump_row_counts]. Canonical tables only; projection / FTS / vec0 shadow tables are excluded. Order matches fathomdb_schema::CANONICAL_TABLES.
DumpSchemaReport: Result of [Engine::dump_schema]. user_version is the PRAGMA user_version sentinel. Canonical tables appear first per fathomdb_schema::CANONICAL_TABLES, then remaining non-sqlite_* tables alphabetically. Indexes follow the same alphabetical rule.
Engine
ExciseRecordReport: 0.8.20 Slice 5b (R-20-E7) — outcome of [Engine::excise_collection_record]. records_excised counts the erased operational_mutations versions (an append-only-log collection keeps every version of a key); state_rows_excised counts the erased operational_state row (0 or 1).
ExciseReport: Phase 9 Pack B excise report (AC-028a/b/c). Counts are post-excise totals; projections_invalidated reports the shadow-row invalidation total (FTS5 + vec0 + projection terminal) for the excised source.
Explanation: 0.8.8 EXP-OBS (Slice 5) — the opt-in retrieval explanation payload returned behind search_explained (the explain=true surface). Built from the engine’s OWN fusion/rerank machinery (fuse_three_arms per-arm ranks, ce_rerank blend components) — no parallel machinery (R-OBS-3). Carries a query-level QueryTrace plus a per-hit breakdown parallel to (and in the same order as) SearchResult.results.
ExtractDocument: G11 (Slice 15) — a document sent to a BYO-LLM extraction harness via Engine::ingest_with_extractor.
Filter: 0.8.11 Slice 40 (#17) — the unified closed Filter contract. ONE superset type with implicit-AND FilterTerms, dispatched to one of two internal compilation backends (Option A — the TYPE unifies, the COMPILATION dispatches): the vec0-metadata indexed pre-KNN WHERE for search_filtered, and json_extract over canonical_nodes.body for read.list. The shipped SearchFilter (G10) and Predicate lists (G4) re-express as sugar that lowers into this type (D4); the filter=None byte-identical-0.7.2-SQL pin is preserved because the vec0 lowering routes back through the shipped vector_filter_clause compilation verbatim.
Finding: Single doctor finding record. Stable report-shape per AC-043c. The code and doc_anchor strings are stable dispatch keys owned by dev/design/recovery.md § Code-to-operator-action cross-reference.
GraphFrontierStats: G0 Phase-2 (E0a / BLOCK-1) — graph-arm frontier instrumentation. A side-channel meter (deliberately NOT a SearchResult/SearchHit field — byte stability) that proves whether the graph arm seeds a non-empty frontier. Under the current doc-seeded path the frontier is empty (doc nodes carry logical_id = NULL), so seeds_resolved == 0 and resolved_seed_rate == 0.0 — this meter is the measurement that proves it (and, post-C1, the 0→>0 flip).
IdSpace: C-2 (0.8.19 / OPP-12 Phase-1, TC-8) — the typed, non-null, id-space-total carrier for SearchHit::id. Subsumes the interim write_cursor id AND the additive Cause-A stable_id field of prior releases: the value is the BARE id (prefix stripped), and to_prefixed reproduces the pre-swap stable_id string byte-for-byte (l:/h: unchanged) so cross-session real-gold keying continues on id as a true no-op.
IngestWithExtractorReceipt: G11 (Slice 15) — receipt returned by Engine::ingest_with_extractor.
IntegrityReport: Three-section integrity report. AC-043a pins exactly these three keys.
MeanRecomputeReport: 0.7.2 PR-2b — result of [Engine::recompute_mean] (the manual doctor recompute-mean path) and of the shared in-transaction recompute core. drift_cos_before is the cosine between the freshly derived corpus mean and the previously-pinned mean (1.0 when nothing was pinned yet, i.e. a first pin). mean_was_pinned distinguishes a refresh of an existing mean from an initial pin. See dev/design/embedder.md §0.3.
NodeRecord: Slice 30 (G2) — an active canonical node row returned by read.get / read.get_many.
OpStoreRow: Slice 30 (G3) — one operational_mutations row returned by read.collection / read.mutations. id is the autoincrement PK (the after-id cursor key).
OpenReport
OpenedEngine
OrphanProvenanceReport: Result of [Engine::orphan_provenance] — the per-source_id census behind fathomdb doctor orphan-provenance (design §4 item 11).
OrphanProvenanceSource: 0.8.20 Slice 5d (R-20-E8) — one source_id bucket in an OrphanProvenanceReport. source_id is None for the NULL-provenance bucket, which after migration step 21 should contain ONLY governed NODES.
PerHitExplain: 0.8.8 EXP-OBS (Slice 5) — per-hit provenance + score breakdown. One entry per returned SearchHit, same order. *_rank is the 0-based rank the hit’s body held in that arm’s pre-fusion list (None = absent from that arm).
ProjectionDelta: 0.8.20 Slice 15d (R-20-PR) — the diff Engine::configure_projections applied. Idempotent re-registration yields unchanged == true with all vecs empty (the “re-registration is a no-op” acceptance signal). A destructive change without an explicit drop is an Err, not a delta.
ProjectionFts: 0.8.20 Slice 15d (R-20-PR) — the searchable→FTS sub-target selector.
ProjectionRuntimeStatus: A pure, current view of projection-runtime facts for one open engine session.
ProjectionRuntimeStatusEntry: One declaration’s current dense status in ProjectionRuntimeStatus.
ProjectionSpec: 0.8.20 Slice 15d (R-20-PR / C-1) — a single declarative projection declaration. HITL-ratified shape (api-surface.md:85-89): { name, roles: Set<ProjectionRole>, fts?, vector? }. roles carries SET semantics (dedup + membership; an attribute can be Filterable AND Searchable) — encoded here as a sorted, de-duplicated BTreeSet. Named roles, not kind (kind is the node/edge type discriminator).
ProjectionVector: 0.8.20 Slice 15d (R-20-PR) — the searchable→vector sub-target selector.
QueryTrace: 0.8.8 EXP-OBS (Slice 5) — query-level retrieval trace. Reuses the existing search_reranked knobs + the active embedder identity; timings are coarse per-stage wall-clock (monotonic) captured only on the explain path.
ReadView: 0.8.20 Slice 10b (R-20-RV / R-20-NV) — the read view: the single knob that decides which canonical_nodes rows a read verb may see.
RebuildReport: Structured result of a rebuild operation. rows_invalidated is the total shadow-state rows truncated before re-derivation; rows_rebuilt is the count of rows the synchronous rebuild loop re-materialised (asynchronous re-enqueue work performed by the projection scheduler is not counted here). projection_cursor_after is the post-rebuild value of the projection cursor.
RecoveryHint: Recovery dispatch surface attached to a corruption detail.
SafeExportArtifact: Result of a successful [Engine::safe_export] call. The returned manifest_sha256 equals the SHA-256 of the export file bytes (per AC-039a) and matches the sha256 field written into the manifest JSON.
SchemaObject: Single table or index entry emitted by [Engine::dump_schema].
SearchExpandResult: Slice 20 (G6) — result of Engine::search_expand: initial search hits plus nodes reached by bounded BFS expansion that are not already in the search hit set.
SearchFilter: G10 — closed metadata filter for Engine::search_filtered (Slice 10).
SearchHit: Derives Clone, Debug, PartialEq but not Eq — score: f64 forbids total equality.
SearchResult: Hybrid search result. results carries structured SearchHits in vector-first, dedup-on-body order. Derives Clone, Debug, PartialEq but not Eq — each hit carries a score: f64.
SoftFallback: Soft-fallback signal carried on hybrid search results.
SourceId: 0.8.20 Slice 5c (R-20-E3) — the provenance of a canonical row: which source document it is attributable to, and therefore what excise_source must erase when that source is withdrawn.
TableRowCount: Single canonical-table row count emitted by [Engine::dump_row_counts].
TraceEvent: Single canonical-row tracing record. table is one of "canonical_nodes" or "canonical_edges".
TraceReport: Phase 9 Pack B trace report (AC-042). One event per canonical row attributable to the requested source_id, ordered by write_cursor ascending.
TruncateWalReport: Result of [Engine::truncate_wal]. Carries the three counters returned by PRAGMA wal_checkpoint(TRUNCATE): busy, log_frames, checkpointed_frames.
VerifyEmbedderReport: Result of [Engine::verify_embedder]. stored_identity is the name:revision pair persisted in _fathomdb_embedder_profiles; supplied_identity echoes the operator’s input verbatim.
WriteReceipt

Enums§

ComparisonOp: G4 (Slice 35) — comparison operator for Predicate::JsonPathCompare.
CorruptionKind: Open-path corruption category.
CorruptionLocator: Locator pointing at the corrupted region of the database file.
DenseReadiness: 0.8.20 Slice 20 (R-20-DR) — the ENGINE-SET readiness of the searchable→vector projection, per dev/design/record-lifecycle-protocol/projection-registry-and-async-embed.md §3.
EmbedderChoice: Caller-facing selector for the embedder used by an opened engine (dev/design/embedder.md §0).
EngineError
EngineOpenError
FilterTerm: 0.8.11 Slice 40 (#17) — a single closed FilterTerm of the unified filter grammar (ADR-0.8.11-filter-grammar-unification, Option A; closes reserved-gap 37). Exactly five variants: the four G10 shorthand metadata fields (SourceType/Kind/CreatedAfter/Status) plus the general G4 json-path Predicate (Json). The shorthand fields are dedicated typed variants — NOT Json(Predicate) over $.source_type etc. — precisely so the vec0 search backend can lower them to the indexed pre-KNN metadata columns while typed-rejecting an arbitrary Json term (D3: no demotion to post-KNN json_extract).
IdSpaceKind: A single structured search hit (G1 / AC-057a-clean).
InitialState: OPP-12 Phase-1 (0.8.19 Slice 5) — the CREATE-TIME subset of LifecycleState.
LifecycleState: OPP-12 record-lifecycle Phase-1 (0.8.19 Slice 5) — the existence axis.
OpenStage: Engine.open stage at which corruption was detected.
Predicate: G4 (Slice 35) — closed typed predicate for Engine::read_list filter.
PreparedWrite: Batch input shape for Engine::write.
ProjectionRole: 0.8.20 Slice 15d (R-20-PR, C-1) — one member of a ProjectionSpec’s role set. Exactly three members (HITL-ratified S8, api-surface.md:87): searchable→FTS and searchable→vector are NOT roles — they are tier labels carried by the fts/vector sub-objects of the spec, so an attribute is Searchable once and the sub-objects select FTS-only / vector-only / both.
ProjectionRuntimeUnavailabilityReason: The reason ProjectionRuntimeStatus::runtime_embedder_available is false.
ProjectionStatusDenseReadiness: The dense-readiness projection of ProjectionRuntimeStatusEntry.
RebuildKind: Which shadow-state surface a RebuildReport describes. Projections covers the full FTS5 + vec0 + projection-terminal rebuild emitted by [Engine::rebuild_projections]. Vec0 covers the vec0-only path emitted by [Engine::rebuild_vec0].
RowKind: EXP-S (0.8.14 Slice 5, D1) — structural-role tag for a canonical row.
ScalarValue: G4 (Slice 35) — scalar value for Predicate comparisons.
Section: One section of an IntegrityReport. Either every check in the section was clean, or one or more typed Findings describe the detected issue. Per AC-043b.
SoftFallbackBranch: Which retrieval branch produced a hit (or could not contribute).
TraversalDirection: Slice 20 (G5) — direction of graph traversal for Engine::graph_neighbors / Engine::search_expand.
TruncateWalStatus: Typed outcome of [Engine::truncate_wal]. Done matches SQLite’s busy = 0 return from PRAGMA wal_checkpoint(TRUNCATE); any other value surfaces as Busy.
VerifyEmbedderStatus: Typed outcome of [Engine::verify_embedder]. Mismatches do not raise EngineError; the operator workflow needs to see the stored vs. supplied pair to decide on next action.

Constants§

DEFAULT_SEARCH_RESULT_LIMIT: Default number of ranked hits returned by public retrieval APIs.
MAX_SEARCH_RESULT_LIMIT: Largest ranked-hit count a public retrieval request may select.
MEAN_VEC_PIN_THRESHOLD: EU-5a2 — number of documents required before the workspace’s _fathomdb_embedder_profiles.mean_vec is pinned for the default profile. Per dev/design/embedder.md §0.3 (compute-once-on-first- ingest lifecycle). Public-visible so the EU-5a2 machinery test can assert the value.
RECENCY_WEIGHT: G12-recency — additive recency weight. Must satisfy two constraints:
RRF_K: G9 — Reciprocal Rank Fusion constant. IR-C (2026-06-10b, performance-output-and-compare.md) found the standard k≈60 slightly too high: the recall gain is concentrated at the top of the list, where a lower k sharpens rank-1/2 contributions. k=30 is the validated operating point (k10 > k30 > k60 > k100 on the sweep, 30 the conservative middle). Fusion is on rank, never raw score.
RRF_WEIGHT_GRAPH: R3 (Slice 30) — graph arm RRF weight. Conservative starting value (equal to RRF_WEIGHT_VECTOR). Without R2 per-class delta data the graph arm weight cannot be calibrated; 1.0 is the minimum non-zero contribution. The graph arm surfaces newly-reachable nodes from BFS traversal; it is not meant to override the primary text/vector signals. Revisable after R2 data arrives. See dev/design/slice-30-design.md §Q2.
RRF_WEIGHT_TEXT
RRF_WEIGHT_VECTOR: G9 / IR-C — per-branch RRF weights. The sweep’s optimum is strongly text-dominant (text:vector ≈ 3:1): the lexical (BM25) arm carries exact-fact recall and the dense arm, over-weighted, is a net drag on exploratory recall (performance-output-and-compare.md, 2026-06-10b/e). A branch contributes weight / (RRF_K + rank).
SEARCH_RERANK_LIMIT: Historical name for the public default ranked-result count (10).
TOP_K_BIT_CANDIDATES

Functions§

apply_importance_reweight: 0.8.16 Slice 5 / F9 — OFF-by-default importance/confidence reweight, applied to the fused hits AFTER bit-KNN + RRF (mirrors [apply_recency_reweight]).
rerank_passages: 0.8.2 Slice E2 — standalone CE rerank of a caller-supplied passage list.