Expand description
A key-value store optimized for atomically committing a small collection of metadata.
Metadata is a key-value store optimized for tracking a small collection of metadata that allows multiple updates to be committed in a single batch. It is commonly used with a variety of other underlying storage systems to persist application state across restarts.
§Format
Data stored in Metadata is serialized as a sequence of key-value pairs in either a “left” or “right” blob:
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| 0 | 1 | ... | 8 | 9 |10 |11 |12 |13 |14 |15 |16 | ... |50 |...|90 |91 |92 |93 |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| Version (u64) | Key1 | Value1 |...| CRC32(u32) |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+To ensure the integrity of the data, a CRC32 checksum is appended to the end of the blob. This ensures that partial writes are detected before any data is relied on.
§Atomic Updates
To provide support for atomic updates, Metadata maintains two blobs: a “left” and a “right” blob. When a new update is committed, it is written to the “older” of the two blobs (indicated by the version persisted). Writes to commonware_runtime::Blob are not atomic and may only complete partially, so we only overwrite the “newer” blob once the “older” blob has been synced (otherwise, we would not be guaranteed to recover the latest complete state from disk on restart as half of a blob could be old data and half new data).
§Delta Writes
If the set of keys and the length of values are stable, Metadata will only write an update’s delta to disk (rather than rewriting the entire metadata). This makes Metadata a great choice for maintaining even large collections of data (with the majority rarely modified).
§Example
use commonware_runtime::{Spawner, Runner, deterministic};
use commonware_storage::metadata::{Metadata, Config};
use commonware_utils::sequence::U64;
let executor = deterministic::Runner::default();
executor.start(|context| async move {
// Create a store
let mut metadata = Metadata::init(context, Config{
partition: "partition".to_string(),
codec_config: ((0..).into(), ()),
}).await.unwrap();
// Store metadata
metadata.put(U64::new(1), b"hello".to_vec());
metadata.put(U64::new(2), b"world".to_vec());
// Sync the metadata store (batch write changes)
metadata.sync().await.unwrap();
// Retrieve some metadata
let value = metadata.get(&U64::new(1)).unwrap();
// Close the store
metadata.close().await.unwrap();
});