Module migration

Expand description

Read-only diagnosis of a store an embedding-model change made unopenable, and the feasibility proof the rebuild depends on (#1762). Never writes to the store it inspects. Read-only diagnosis of a store that a changed embedding model has made unopenable, and the feasibility proof the rebuild depends on (#1762, PR A).

§What this module is NOT

It does not migrate, does not switch anything over, and never writes to the store it inspects. Producing a crate::migration::MigrationState is a later step behind an explicit prepare command; a diagnosis yields a crate::migration::DiagnosisReport and nothing else.

§Why a feasibility proof comes first

A rebuild must re-insert every fact under its ORIGINAL u64 id: edges are (id, from, to, relation) with no vector of their own, entity hubs derive their id from the topic, and the working-context index addresses facts by id. Renumbering would silently sever all three. So before any rebuild code is written, the architecture has to be shown to support reading every fact back out — ids, content, ordinary metadata, RESERVED metadata and the absolute expiry — and putting it back unchanged.

MemoryStore offers no enumeration at all: every read is by id or a top-k vector search, and count() counts without listing. Two paths down into the engine do, and they are not equivalent:

a VelesQL scan with no vector predicate, walked by LIMIT/OFFSET (crate::migration::enumerate_collection) — complete and deterministic, but quadratic, and BOUNDED: the pipeline clamps limit + offset to 100_000 and goes silently empty past that mark;
the collection’s own scroll_batch (crate::migration::enumerate_by_cursor) — a cursor keyed on the point id, exclusive and ascending, which bypasses the query pipeline and so carries neither the clamp nor the re-sort.

The first was written first because WHERE id > n genuinely does not work — filters read the payload and the id is not in it. That ruled out expressing a cursor in VelesQL; it did not rule out the cursor, and treating the query language’s limit as the architecture’s limit is the error this module now records rather than repeats.

That either parses is not the proof. Both are measured by running them against a seeded store and comparing what comes back, field by field, and against each other.

Structs§

BatchReinsertion: What a batch re-insertion produced.
CollectionInventory: One collection as the rebuild will find it.
DiagnosisReport: Everything a rebuild needs to know about a store, and nothing it could act on by accident.
EdgeReinsertion: What putting the edges back produced.
ExecuteOutcome: What one execute run did, and where its artefacts live.
MigrateOptions: What the operator asked the command to do.
MigrateOutcome: What one migrate run did. The early stages are None exactly when the journal routed past them — reporting a rebuild that did not run would be misreporting, and with the source already archived it could not have run.
MigrationLock: Exclusive possession of a migration workspace.
MigrationState: What a prepared migration recorded, so a later run can decide whether to resume it.
RawFact: One fact as the rebuild will need to re-create it.
RebuildDestination: The destination store, created and sized by the caller.
RebuildJournal: Where the journal lives and the proof we may write it.
RebuildOutcome: What a completed pass did — counts, not verdicts. The verdict is the destination re-reads performed along the way.
RebuildSource: The source store, opened read-only in spirit: nothing here writes to it.
SwitchOutcome: What a completed switch did.
SwitchState: Which of the three directories exist when a switch-over is interrupted.
TargetContract: What the operator is pointing the rebuild at: the target embedder’s identity, and the regime they selected.
TtlSummary: What the expiries in a collection amount to.
ValidationOutcome: What one validation pass established.

Enums§

Capability: Whether a capability the rebuild depends on is established, or missing.
CollectionProgress: How far one collection’s rebuild got inside Phase::Prepared.
Compatibility: What the source’s own record permits, independently of what was asked for.
Phase: Where a migration has got to.
Recovery: What to do with a migration found stopped.
Reinsertion: What putting a fact back produced.
Resolution: What the rebuild will do, or why it will not run.
SourceProvenance: What the store itself records about the embedder that filled it.
Strategy: What the operator selected on the command line.
VectorPolicy: Which vector each reinserted fact carries.

Constants§

AGENT_COLLECTIONS: Collections AgentMemory opens, all at the same dimension — so any one of them refusing the new dimension makes the whole store unopenable, and an inventory that skipped the empty ones would under-report the work.
ARCHIVE_SUFFIX: The archive slot: a sibling of the source, named after it.
DIAGNOSIS_FORMAT_VERSION: The shape of a DiagnosisReport, stamped into every report.
LOCK_FILE: The file that marks a migration in progress.
PHASES: Every phase, in order — so an exhaustive check cannot silently miss one added later.
STATE_FILE: The file a prepared migration records its state in.
STATE_FORMAT_VERSION: The shape of a MigrationState.
STATE_TEMP_FILE: The fixed sibling staging file for an atomic state replacement.

Functions§

assess: Read the source’s record against the target contract.
bytes_on_disk: Sum of every regular file’s length under root.
cross_check_edges: The same collection of edges, gathered through the INCOMING index instead.
default_scratch_parent: The default scratch parent: the directory the store itself sits in.
diagnose: Inspect source and report what a rebuild onto target_model would face.
dry_run: Diagnose the store under options against target, and render the result.
enumerate_by_cursor: Read every fact of collection out of db by cursor, in batches of batch.
enumerate_collection: Read every fact of collection out of db, in pages of page.
enumerate_page: One page of collection, starting at offset — the unit a checkpoint resumes from, and what makes the walk above interruptible rather than all-or-nothing.
execute: Diagnose, stage, lock, rebuild, release — the whole non-dry-run path.
export_edges: Every edge of collection that lies between two exported facts, as complete tuples.
export_edges_verified: The export, checked against the incoming index over ONE snapshot of the live facts.
fingerprint: A versioned SHA-256 digest of every directory, regular file and file byte.
migrate: Rebuild, validate and switch — entering wherever the journal stands.
migration_complete_notice: Re-exported so the binary can print the completion without duplicating it.
parse_migrate_args: Parse migrate-embeddings’ flags.
rebuild: Run the rebuild to completion, resuming from whatever state records.
refuses: Whether this diagnosis leaves the command with nothing it could run.
reinsert: Put fact back into collection under its ORIGINAL id, with vector as the caller decided it.
reinsert_batch: Put a whole batch back, in one write.
reinsert_edges: Put edges back into collection, AFTER the facts.
render: Render a report for an operator: what is here, what would happen, and what still blocks it.
require_destination: Require the flag a non-dry-run cannot proceed without.
resolve: Decide the regime from what was asked and what the store permits.
same_filesystem: Whether a and b sit on the same filesystem.
scroll_page: One batch of collection, starting strictly after cursor, with the cursor to resume from.
switch_over: Drive the switch from wherever the journal stands to Committed.
validate_destination: Validate destination against store and journal the result.