SQLite schema definition + migration ladder. v0.7.0 L0.5-3
extracted the SCHEMA constant, the MIGRATION_V*_SQLITE
include-bytes constants, the CURRENT_SCHEMA_VERSION parallel
constant, and the migrate function out of src/db.rs into
this sub-module. Pure refactor — semantics unchanged. The
MAX_SUPPORTED_SCHEMA constant in cli::boot must still bump
in lockstep with [CURRENT_SCHEMA_VERSION] (current value: 57).
Versions 45/46 are reserved for sibling provenance-write landings
(Gaps 1+2, #884/#885); this crate jumps 44 → 47 for Gap 3 (#886).
v48 (Track D #933) adds the federation_push_dlq table so quorum-
broadcast fanout failures can be replayed by the new
replay_federation_push_dlq worker.
v51 (#1255) adds the federation_nonce_cache table so the
FederationNonceCache LRU persists across daemon restarts —
pre-#1255 every restart opened a fresh replay window for any
(body, sig, nonce) tuple captured before the restart.
v52 (#1389) adds the transcript_line_dedup table backing the
sha256-keyed idempotency layer for the four-layer capture
architecture (L2 recover-on-boot + L3 substrate watcher + L4
memory_capture_turn MCP tool). Closes the #1388 substrate
failure mode at the storage layer.
Phase P6 — outcome of applying a token budget to a ranked recall
list. Carries everything mcp::handle_recall needs to populate the
new RecallMeta block (budget_tokens_used, budget_tokens_remaining,
memories_dropped, budget_overflow).
Maximum sync-clock skew in seconds across the sync_state table —
the largest gap between last_pulled_at (when this peer last heard
from a peer) and last_seen_at (the peer’s own updated_at advance).
Returns Ok(None) when sync_state is empty or the columns are
missing on a pre-T3 schema.
Typed error returned by insert_with_conflict under
ConflictMode::Error when a (title, namespace) row already
exists. Carries the existing row’s id so callers can surface a
well-shaped diagnostic instead of leaking a generic SQL string.
v0.6.3.1 P2 (G4): error returned by set_embedding when a write would
introduce a new embedding dimensionality into a namespace that has already
established one via an earlier write. Surfaced as a typed error so the
MCP/HTTP handlers can map it to a 409 Conflict rather than letting cosine
silently return 0.0 on every subsequent recall.
Typed substrate-layer marker error for the pre-write hook refusal
path. Wrapped in anyhow::Error so the existing
anyhow::Result<String> return shape of storage::insert* stays
unchanged — the handler layer downcasts via
MemoryError::from(anyhow::Error) (see src/errors.rs) to map
the refusal to HTTP 403 FORBIDDEN + code GOVERNANCE_REFUSED.
Outcome of invalidate_link (Pillar 2 / Stream C —
memory_kg_invalidate). valid_until is the timestamp now stored on
the link; previous_valid_until is the prior value, or None if
this was the first invalidation. Callers can use the prior value to
distinguish a fresh supersession from an idempotent retry.
v0.7.0 recursive-learning Task 6/8 — optional in-substrate hook
callbacks fired by reflect_with_hooks. Bundled into a single
struct so the substrate signature stays compact and so future
callbacks (e.g. on-rollback) can land without churning every
call site.
Input bundle for reflect. Holds every caller-tunable field of the
new reflection memory plus the source-id list. Defaults mirror the
MCP tool schema (tier=mid, priority=5, confidence=1.0,
source=DEFAULT_NHI_SOURCE per crate::validate::DEFAULT_NHI_SOURCE
= "nhi" post-#1175 — pre-#1175 this defaulted to "claude", a
heterogeneous-NHI monoculture defect that #1175 closed) so the
dispatch layer can build this from the raw JSON arguments without
further fixup.
Update a memory by ID. Returns (found, content_changed) so callers can
re-generate embeddings when the searchable text has changed.
v0.7.0 Provenance Gap 1 (issue #884) — typed optimistic-concurrency
error returned by update_with_expected_version when the caller
passed expected_version and the stored row’s current version
has drifted. Carries both expected + current so the caller can
surface a useful diagnostic and choose between re-read+re-apply
or bubbling CONFLICT upstream.
Identifies which end of a link a missing-memory refusal refers to.
None is reserved for memory-not-found errors that are not part of
a link operation. The Source and Target variants preserve the
pre-#962 user-facing error prefixes (“source memory not found: …” /
“target memory not found: …”) so existing string-matching consumers
keep working through the typed enum’s Display impl.
Typed substrate-level error surface for reflect. Kept distinct
from crate::errors::MemoryError so the SQLite substrate layer
stays free of HTTP-status concerns; the caller at the MCP / HTTP
boundary maps these into the wire-shaped variant. Task 5/8 matches
on ReflectError::DepthExceeded here (and the equivalent
MemoryError::ReflectionDepthExceeded variant) to emit the
signed_events audit record for the refusal decision.
Typed substrate-layer error categories. Each variant maps to a
canonical HTTP status via MemoryError::from(anyhow::Error) and
preserves the original bail!() message verbatim via Display so
downstream .to_string().starts_with(...) and .contains(...)
consumers keep working through the typed layer.
Default page size for archive listings (HTTP /api/v1/archive and
MCP memory_archive_list) when the caller passes no explicit
limit — one knob so both surfaces page identically.
#1558 batch 5 wave 3 — canonical source value stamped on rows
minted by consolidate (MCP memory_consolidate + the HTTP
power-consolidation handler pass it verbatim). Listed in
validate::VALID_SOURCES; one spelling, hoist-only.
Default cosine similarity threshold for declaring a candidate a
duplicate. Empirically tuned for MiniLM-L6-v2 (the local embedder):
near-paraphrases of the same memory tend to land at 0.88+, while
loosely related content sits well below 0.85. Callers can override.
Hard floor for duplicate-check threshold. Below this, anything can match
random unrelated content — refuse to honor the lookup so callers don’t
silently get garbage merge suggestions.
Hard ceiling on traversal depth supported by find_paths.
Distinct from KG_QUERY_MAX_SUPPORTED_DEPTH because path
enumeration is more expensive than reachability — we can afford a
slightly deeper budget for the BFS but not by much.
Default cap on rows returned by kg_query when the caller does not
specify one (Pillar 2 / Stream C). Mirrors kg_timeline’s default so
the two traversal tools behave consistently for agents driving them.
Maximum traversal depth supported by kg_query. The recursive-CTE
implementation enforces an explicit ceiling so a crafted call cannot
run an unbounded traversal; the charter (v0.6.3-grand-slam.md
§ Performance Budgets) sets the published budget at depth ≤ 5.
Default cap on rows returned by kg_timeline when the caller does
not specify one (Pillar 2 / Stream C). Sized to fit a reasonable
agent context window without paging — callers needing more should
pass an explicit limit.
Error prefix emitted when validate_link_pre_create rejects a
reflects_on edge that would close a cycle in the reflection graph.
HTTP / SAL response mappers look for this prefix to surface 409
CONFLICT; MCP surfaces it as a plain text error. Centralised so all
three entry points stay in lockstep with StorageError::LinkReflectionCycle.
Error prefix emitted when the K9 permission pipeline returns Deny
for a link write. HTTP / SAL response mappers translate this to 403
FORBIDDEN. Paired with StorageError::LinkPermissionDenied.
Default row cap for memory list/search surfaces when the caller
passes no explicit limit. Mirrored by the postgres SAL adapter
(src/store/postgres.rs::list_by_source_uri) so both backends
page identically.
Post-clamp usize → i64 conversion fallback for list/query limits.
Unreachable in practice (values are already clamped to at most
LIST_MAX_LIMIT, which always fits i64); kept as a named knob so
the fallback page size is explicit rather than magic.
Hard ceiling on rows returned by the memory list/search surfaces.
One shared knob across the sqlite + postgres SAL adapters; same
family as KG_TIMELINE_MAX_LIMIT / KG_QUERY_MAX_LIMIT.
#1579 A5 — minimum Jaccard token overlap between the incoming
content and a cosine-near-duplicate candidate’s content for the
pair to be classified as a proactive conflict.
#1579 A5 — k requested from the HNSW index by
proactive_conflict_check_with_index. Deliberately larger than
PROACTIVE_CONFLICT_TOP_K because the index is global while the
conflict check is namespace-scoped: the namespace filter is applied
AFTER the ANN search (post-filter semantics), so foreign-namespace
hits consume slots. 32 gives the in-namespace pool ample headroom
(the ≥ 0.95 cosine gate means only near-identical vectors matter,
and > 32 near-identical foreign-namespace rows crowding out an
in-namespace conflict is a pathology the bounded fallback’s
advisory contract already tolerates — see
PROACTIVE_CONFLICT_SCAN_LIMIT).
Hard cap on input groups walked when assembling a taxonomy tree.
Even when callers pass a wildly large limit, we never walk more
than this many (namespace, count) rows — bounds memory + time.
Shared by the sqlite + postgres taxonomy paths and the HTTP / MCP
taxonomy surfaces so all four clamp identically.
Optional governance pre-write hook. When Some, every substrate
INSERT path consults the closure BEFORE the SQL write; an
Err(reason) short-circuits the write with no row touched.
Phase P6 (R1) — context-budget greedy fill. Iterates over scored
candidates in rank order; stops at the first memory whose inclusion
would exceed the budget — UNLESS the output is still empty, in
which case the highest-ranked memory is returned anyway with
budget_overflow = true. This preserves the R1 guarantee that a
successful recall always returns at least one result when any
matched, even if the user supplied an unrealistically tight budget.
Move a memory from memories to archived_memories. Used by the
HTTP /api/v1/archive explicit-archive endpoint (S29) and by
sync_push when a peer pushes an archives: [id] record.
#940 (security-high, 2026-05-20) — caller-scoped archive variant.
Mirrors archive_memory but constrains the soft-move to rows
in the live memories table whose metadata->'agent_id' JSON
field matches caller (with the inbox-target carve-out:
metadata->'target_agent_id' == caller is also archivable by
the inbox owner, matching
[crate::store::is_visible_to_caller]).
Build the namespace inheritance chain in top-down order
(["*", root, ..., leaf]). Mirrors and replaces the historical
mcp::build_namespace_chain so non-MCP call sites (db-layer
governance enforcement, HTTP handlers, future hook pipelines) can
reuse the same walk.
Canonical hash used by check_duplicate_with_text to detect
byte-identical title + content pairs even when the embedding
pipeline (lower-casing, prefix tagging, etc.) prevents the cosine
similarity from saturating at 1.0.
v0.7.0 #1416 / RFC-0001 — sqlite SSOT for the L4 layered-capture
idempotent write. Both the MCP memory_capture_turn handler (which
holds a raw &rusqlite::Connection) and SqliteStore:: capture_turn_idempotent (the SAL trait surface) call through here,
so the dedup-lookup + atomic three-row insert exists in exactly one
place on the sqlite path.
v0.6.3 (capabilities schema v2): count namespace standards whose
metadata.governance is non-null. A “rule” here means a namespace
has an explicit governance policy attached to its standard memory.
The count is a transparent passthrough — the full permission system
arrives in v0.7 (arch-enhancement-spec §3).
#1579 B3 — count of rows carrying a stored embedding. Cheap probe
(no blob decode, no row materialisation) used by the CLI recall
path to decide whether a one-shot invocation should pay the HNSW
graph-construction cost at all (see
crate::hnsw::CLI_HNSW_BUILD_MIN_ENTRIES).
Phase P6 — token cost of a memory’s content only (not title), per
the R1 spec which budgets against the LLM context window. Title and
metadata are caller-side ornament; content is what gets stuffed
into the prompt.
v0.6.3 (capabilities schema v2): count pending_actions rows whose
status matches the predicate. Used by handle_capabilities to
surface live approval queue depth.
v0.6.3 (capabilities schema v2): count rows in the subscriptions
table. Used by handle_capabilities as a proxy for “registered
hooks” — the hook pipeline itself is v0.7 Bucket 0 work.
Phase P6 (R1) — count tokens in text using OpenAI’s cl100k_base
BPE encoding. This is the de-facto standard for Claude / GPT context
budgeting and is shipped with tiktoken-rs (the BPE table is embedded
in the crate, ~1.7 MB, so the count is offline-deterministic across
all hosts). The encoder is built lazily and cached process-wide via
OnceLock — cl100k_base() itself parses the embedded table on every
call, which adds a few ms; we pay that cost once.
Mark a pending action as approved or rejected. Returns true on status
transition. Does NOT execute the action itself — the caller replays
the payload on approval (the db layer doesn’t know how to execute
cross-interface write semantics).
Count rows whose stored embedding_dim does not match what the BLOB
contains (or where the column is missing while a BLOB exists). Surfaced
in Stats::dim_violations and consumed by P7 doctor.
#1598 — distinct embedding dimensionalities currently stored,
optionally namespace-filtered, for the reembed pre-flight banner
(the loud “old dims vs target dim” disclosure before a vector-space
migration). Prefers the declared embedding_dim column and falls
back to deriving from the BLOB length for legacy rows — 4n+1
bytes is the v17 headed form ((len-1)/4 floats), 4n the
legacy unheaded form (len/4), mirroring dim_violations.
Count rows whose embedding_dim (post-P2) does not match the modal
dim within their namespace. On pre-P2 schemas the embedding_dim
column doesn’t exist; the function returns Ok(None) so the doctor
can render “not yet observed (pre-P2 schema)”.
Count of namespaces that have a standard registered with a non-null
metadata.governance block, and the count without (just a standard
memory but no policy attached).
Distribution of the parent_namespace chain depth across
namespace_meta rows. Returns a Vec where index i is the count of
namespaces with chain depth i (depth 0 = no parent).
Age in seconds of the oldest pending row in pending_actions, or
None if the queue is empty (or the column is unparseable). The
doctor uses this to flag a backlog older than 24h as critical.
Sum of subscriptions.dispatch_count and subscriptions.failure_count
across all rows. Returns (dispatched, failed). Used by the doctor to
estimate webhook delivery success rate.
#1598 — (total_rows, rows_with_embeddings) for the reembed
dry-run plan, optionally namespace-filtered. COUNT(embedding)
counts non-NULL values, so the missing count is the difference.
Enforce governance for a GovernedAction. On GovernanceDecision::Pending,
a row is inserted into pending_actions and the returned pending_id is
embedded in the decision.
Phase P6 — kept for backward compatibility with the Task 1.11 byte-
heuristic surface. New code should use count_memory_tokens. The
returned value is now BPE-accurate (cl100k_base) rather than the
prior len/4 estimate, so callers reading this through the public
API get the more accurate value automatically.
Task 1.10 — Execute an approved pending action’s payload. Callers invoke
this after approve_with_approver_type returns Approved. Returns the
affected memory id (new id for store, existing id for delete/promote).
v0.6.3.1 P2 (G6) — quick existence check for (title, namespace). Used by
on_conflict='error' callers to short-circuit before the full upsert
machinery runs. Returns the existing row id if there is one.
Look up a memory by ID prefix. Returns the memory if exactly one match is found.
Returns Ok(None) if no matches. Returns an error if the prefix is ambiguous (>1 match).
Fetch the single link identified by the (source_id, target_id, relation)
composite primary key — the only unique identifier memory_links
exposes today.
#1598 — keyset-paginated scan over ALL live memories (embedded or
not), optionally namespace-filtered, for the ai-memory reembed
full-corpus sweep. Same cursor semantics as
get_unembedded_ids_batch_after: at most limit(id, title, content) triples with id strictly after after_id, in id
order. Four distinct prepared shapes (namespace × cursor) keep the
scan sargable (v55/v56 discipline).
v0.6.2 (S35): read the full namespace_meta row for a namespace so the
caller can fan it out to peers. Returns None when no standard is set.
Mirrors the (namespace, standard_id, parent_namespace, updated_at)
tuple used by set_namespace_standard.
Insert with timestamp-aware conflict resolution for sync.
Only overwrites if the incoming memory is newer (by updated_at,
tiebroken by memory.id for a total order across peers —
ultrareview #344, #345).
Mark a KG link as superseded by setting its valid_until column
(Pillar 2 / Stream C — memory_kg_invalidate). Returns Ok(None)
when the (source_id, target_id, relation) triple does not match an
existing link. The supplied valid_until defaults to the current
wall-clock time in RFC3339 form when omitted; callers needing
historical or future supersession can pass an explicit value.
Outbound KG traversal from a source memory (Pillar 2 / Stream C —
memory_kg_query). Returns one row per link reachable within
max_depth hops, filtered by:
Ordered fact timeline for an entity (Pillar 2 / Stream C —
memory_kg_timeline). Returns outbound assertions from
source_id, ordered by valid_from ASC and tie-broken by
created_at ASC for deterministic display.
v0.7.0 K5 — enumerate every namespace whose standard memory carries an
explicit metadata.governance policy and return (namespace, policy)
pairs sorted lexicographically by namespace.
v0.7.0 Provenance Gap 6 (issue #889) — list every memory carrying
the supplied source_uri. Bypasses the FTS layer so callers that
want the full reciprocal set (“every memory from this document”)
don’t need to type a query. Hits the partial
idx_memories_source_uri index directly. Pure read.
Return memories whose updated_at > since, ordered by updated_at
ascending. Used by GET /api/v1/sync/since to stream incremental
updates to a peer. Caps at limit rows (caller-chosen pagination).
#1579 A5 — verify an ANN-derived candidate id list against the DB
and apply the conflict verdict. Fetches only the named rows (point
lookups by PK), re-applies the live/namespace filters the table
scan used, and recomputes EXACT cosine from the stored embedding
blob so the decision function is identical to the scan path.
Task 1.12 — proximity boost applied to a memory’s score based on its
depth distance from the queried agent namespace. Uses the formula
1 / (1 + depth_distance * 0.3) per spec. Distance 0 = full strength
(1.0), each step up the hierarchy dampens linearly.
#936 (security-critical, 2026-05-20) — caller-scoped purge variant.
Mirrors purge_archive but constrains the DELETE to rows whose
metadata->'agent_id' JSON field matches caller (with the
inbox-target carve-out: rows whose metadata->'target_agent_id'
matches caller are also purgeable by the inbox owner, matching
the SAL [crate::store::is_visible_to_caller] visibility
predicate).
Hybrid recall — FTS5 keyword search + semantic cosine similarity.
Returns memories ranked by a blended score of keyword and semantic relevance.
When an HNSW vector_index is provided, uses approximate nearest-neighbor
search instead of scanning all embeddings linearly.
v0.6.3.1 (P3): hybrid recall preserving the existing 2-tuple return
shape for HTTP / CLI / bench callers. Delegates to
recall_hybrid_with_telemetry and discards the telemetry. Kept so
the dozen-plus call sites need no churn for a feature only MCP
handle_recall consumes.
FX-4 / PERF-2 (2026-05-26) — convenience wrapper for the HTTP
recall handler. Same return shape as recall_hybrid but accepts
a pre-computed HNSW hit slice (caller ran idx.search() outside
the DB lock) so the DB-mutex hold window does not cover the
CPU-bound ANN walk. Telemetry is dropped on this path; the HTTP
surface does not consume it today.
FX-4 / PERF-2 (2026-05-26) — variant of
recall_hybrid_with_telemetry that accepts a pre-computed slice
of HNSW hits in place of the in-pipeline idx.search() call. The
HTTP recall handler runs the ANN walk OUTSIDE the DB mutex (the
HNSW index lives behind its own vector_index mutex; the DB lock
is not required for the search) and passes the result here so the
DB-mutex hold window covers only the FTS5 query + the batched
get_many fetch + the touch ops. Concurrent recalls overlap
their CPU-bound ANN walks instead of serialising behind the
single shared connection.
Recall — fuzzy OR search + touch + auto-promote + TTL extension.
Task 1.11: after ranking, applies optional budget_tokens cap.
Phase P6: returns the full BudgetOutcome (tokens_used,
tokens_remaining, memories_dropped, budget_overflow) instead of just
the prior bare tokens_used. Callers that only need tokens_used
read outcome.tokens_used.
v0.6.3.1 (P3): keyword-only recall with retrieval-stage telemetry.
Record a capability-expansion attempt. Used by
handle_capabilities_family after the allowlist decision is made.
Records BOTH grant and deny outcomes so operators can see attempted
access patterns even when the gate refused.
#940 (security-high, 2026-05-20) — caller-scoped restore variant.
Mirrors restore_archived but constrains the INSERT-SELECT to
rows whose metadata->'agent_id' JSON field matches caller
(with the inbox-target carve-out: rows whose
metadata->'target_agent_id' matches caller are also
restorable by the inbox owner, matching the SAL
[crate::store::is_visible_to_caller] visibility predicate).
v0.7.0 Provenance Gap 6 (issue #889) — search with optional
reciprocal source_uri filter. When source_uri is Some(uri),
the FTS search is post-filtered (in SQL) to memories whose
source_uri column equals the supplied value verbatim. The
partial idx_memories_source_uri index (created at v38) covers
the lookup, keeping it O(log N) over the URI-keyed subspace.
Seed the process-wide mmap size for every subsequent open.
Idempotent — first writer wins; later calls are no-ops (matches
crate::quotas::set_quota_defaults).
Look up this peer’s last-push watermark for peer_id. Returns None
if we’ve never successfully pushed to them (foundation-era rows also
return None because the column was added in schema v12).
Record the latest updated_at this local agent has observed from peer_id.
Monotonic by timestamp — older writes do not overwrite newer ones.
Lazily creates the row on first observation.
v0.7.0 H6 (round-2) — truncate a DateTime<Utc> to microsecond
precision. Companion of the same-named helper in
store/postgres.rs:3539 (G3 fix); both ends of the link sign/verify
roundtrip now collapse sub-microsecond digits BEFORE CBOR
canonicalisation. PostgreSQL’s TIMESTAMPTZ stores microseconds —
the SQLite path was lossless, but a link created on SQLite and
later re-verified on Postgres (or vice versa via federation) would
see the canonical RFC3339 string change shape on the storage hop
and break the Ed25519 signature. Truncating at write time makes the
shape stable across adapters. See store/postgres.rs:3520-3543 for
the full design context.
v0.7.0 Provenance Gap 5 (issue #888) — append-and-archive write
path. Used by the MCP memory_update tool when the caller passes
edit_source of llm or hook. Atomic: every step runs inside
a BEGIN IMMEDIATE / COMMIT pair so a failure mid-way leaves
the old row live (no partial supersede).
v0.7.0 Provenance Gap 1 (issue #884) — optimistic-concurrency aware
variant of update. When expected_version is Some(v), the
update fails with a typed VersionConflict error if the stored
row’s version is not equal to v. When None, the legacy
last-write-wins behaviour is preserved (still bumps version on
success). On a successful mutation the row’s version is
monotonically incremented; the new value is observable on the
subsequent read.
v0.6.2 (S34): upsert a pending_actions row from a canonical PendingAction
struct — used by sync_push to apply a peer-originated pending row so
governance state is cluster-consistent. Preserves approvals and
decision fields verbatim so re-plays converge. Uses INSERT ... ON CONFLICT(id) DO UPDATE because the originator’s id is stable across
peers (unlike queue_pending_action which mints a fresh UUID per
queue call).