Module exonum_merkledb::migration

Migration utilities.

Stability

The entirety of this module is considered unstable. While the supported functionality is unlikely to break, the implementation details may change in the following releases.

Migration workflow

Migration refers to the ability to update data in indexes, remove indexes, change index type, create new indexes, and package these changes in a way that they can be atomically committed or rolled back. Accumulating changes in the migration, on the other hand, can be performed iteratively, including after a process shutdown.

Each migration is confined to a namespace, defined in a similar way as Prefixed accesses. For example, namespace test concerns indexes with an address starting with test., such as test.foo or (test.bar, 1_u32), but not test or test_.foo. The namespace can be accessed via Migration.

Migration is non-destructive, i.e., does not remove the old versions of migrated indexes. Instead, new indexes are created in a separate namespace. For example, index foo in the migration namespace test and the original test.foo index can peacefully coexist and have separate data and even different types. The movement of data is performed only when the migration is finalized.

Retaining an index in the migration is a no op. Removing an index is explicit; it needs to be performed via create_tombstone method. Although tombstones do not contain data, they behave like indexes in other regards. For example, it is impossible to create a tombstone and then create an ordinary index at the same address, or vice versa.

A migration can also store temporary data in a Scratchpad. This data will be removed when the migration is finalized.

Indexes created within a migration are not aggregated in the default state hash. Instead, they are placed in a separate namespace, the aggregator and state hash for which can be obtained via respective Migration methods.

It is possible to periodically persist migrated data to the database (indeed, this is a best practice to avoid out-of-memory errors). It is even possible to restart the process handling the migration, provided it can recover from such a restart on the application level. To assist with fault tolerance, use persistent iterators.

Finalizing Migration

To finalize a migration, one needs to call flush_migration. This will replace old index data with new, remove indexes marked with tombstones, and return migrated indexes to the default state aggregator. To roll back a migration, use rollback_migration. This will remove the new index data and corresponding metadata. Both flush_migration and rollback_migration will remove the Scratchpad associated with the migration.

Examples

let db = Arc::new(TemporaryDB::new());
// Create initial data in the database.
let fork = db.fork();
fork.get_list("test.list").extend(vec![1_u32, 2, 3]);
fork.get_proof_entry("test.entry").set("text".to_owned());
fork.get_map(("test.group", &0_u8)).put(&1, 2);
fork.get_map(("test.group", &1_u8)).put(&3, 4);
db.merge(fork.into_patch())?;
let initial_state_hash = SystemSchema::new(&db.snapshot()).state_hash();

// Create migration helper.
let mut migration = MigrationHelper::new(Arc::clone(&db) as Arc<dyn Database>, "test");
{
    // Merkelize the data in the list.
    let old_list = migration.old_data().get_list::<_, u32>("list");
    let new_data = migration.new_data();
    new_data.get_proof_list("list").extend(&old_list);
}

// It is possible to merge incomplete changes to the DB.
migration.merge()?;
// Changes in the migrated data do not influence the default state hash.
let snapshot = db.snapshot();
let intermediate_state_hash = SystemSchema::new(&snapshot).state_hash();
assert_eq!(intermediate_state_hash, initial_state_hash);
// Instead, they influence the state hash for the migration namespace
// (i.e., `test` in this case).
let aggregated = Migration::new("test", &snapshot).state_aggregator();
assert!(aggregated.contains("test.list"));
assert!(!aggregated.contains("test.entry"));

// Leave `test.entry` in place (this is no op).
// Remove one of indexes in `test.group`.
migration.new_data().create_tombstone(("group", &0_u8));
// Create a new index.
migration.new_data().get_proof_entry("other_entry").set("other".to_owned());

// Finish the migration logic.
let migration_hash = migration.finish()?;
// For now, migrated and original data co-exist in the storage.
let snapshot = db.snapshot();
assert_eq!(snapshot.get_list::<_, u32>("test.list").len(), 3);
let migration = Migration::new("test", &snapshot);
assert_eq!(migration.get_proof_list::<_, u32>("list").len(), 3);

// The migration can be committed as follows.
let mut fork = db.fork();
flush_migration(&mut fork, "test");
db.merge(fork.into_patch())?;
let snapshot = db.snapshot();
assert_eq!(snapshot.get_proof_list::<_, u32>("test.list").len(), 3);
assert_eq!(
    snapshot.get_proof_entry::<_, String>("test.other_entry").get().unwrap(),
    "other"
);

Structs

AbortHandle	Handle allowing to signal to MigrationHelper that the migration has been aborted. Signalling is performed on handle drop, unless it is performed with forget method.
Migration	Access to migrated indexes.
MigrationHelper	Migration helper.
PersistentIter	Persistent iterator that stores its position in the database.
PersistentIters	Factory for persistent iterators.
PersistentKeys	Persistent iterator over index keys that stores its position in the database.
Scratchpad	Access to temporary data that can be used during migration. The scratchpad is cleared at the end of the migration, regardless of whether the migration is successful.

Enums

MigrationError

Errors emitted by MigrationHelper methods.

Functions

flush_migration	Flushes the migration to the fork. Once the fork is merged, the migration is complete.
rollback_migration	Rolls back the migration.