Module matterdb::migration

Migration utilities.

Stability

The entirety of this module is considered unstable. While the supported functionality is unlikely to break, the implementation details may change in the following releases.

Migration workflow

Migration refers to the ability to update data in indexes, remove indexes, change index type, create new indexes, and package these changes in a way that they can be atomically committed or rolled back. Accumulating changes in the migration, on the other hand, can be performed iteratively, including after a process shutdown.

Each migration is confined to a namespace, defined in a similar way as Prefixed accesses. For example, namespace test concerns indexes with an address starting with test., such as test.foo or (test.bar, 1_u32), but not test or test_.foo. The namespace can be accessed via Migration.

Migration is non-destructive, i.e., does not remove the old versions of migrated indexes. Instead, new indexes are created in a separate namespace. For example, index foo in the migration namespace test and the original test.foo index can peacefully coexist and have separate data and even different types. The movement of data is performed only when the migration is finalized.

Retaining an index in the migration is a no op. Removing an index is explicit; it needs to be performed via create_tombstone method. Although tombstones do not contain data, they behave like indexes in other regards. For example, it is impossible to create a tombstone and then create an ordinary index at the same address, or vice versa.

A migration can also store temporary data in a Scratchpad. This data will be removed when the migration is finalized.

Indexes created within a migration are not aggregated in the default state hash. Instead, they are placed in a separate namespace, the aggregator and state hash for which can be obtained via respective Migration methods.

It is possible to periodically persist migrated data to the database (indeed, this is a best practice to avoid out-of-memory errors). It is even possible to restart the process handling the migration, provided it can recover from such a restart on the application level. To assist with fault tolerance, use persistent iterators.

Finalizing Migration

To finalize a migration, one needs to call flush_migration. This will replace old index data with new, remove indexes marked with tombstones, and return migrated indexes to the default state aggregator. To roll back a migration, use rollback_migration. This will remove the new index data and corresponding metadata. Both flush_migration and rollback_migration will remove the Scratchpad associated with the migration.

Examples

None yet.

Structs

AbortHandle	Handle allowing to signal to MigrationHelper that the migration has been aborted. Signalling is performed on handle drop, unless it is performed with forget method.
Migration	Access to migrated indexes.
MigrationHelper	Migration helper.
PersistentIter	Persistent iterator that stores its position in the database.
PersistentIters	Factory for persistent iterators.
PersistentKeys	Persistent iterator over index keys that stores its position in the database.
Scratchpad	Access to temporary data that can be used during migration. The scratchpad is cleared at the end of the migration, regardless of whether the migration is successful.

Enums

MigrationError

Errors emitted by MigrationHelper methods.

Functions

flush_migration	Flushes the migration to the fork. Once the fork is merged, the migration is complete.
rollback_migration	Rolls back the migration.