Expand description
An immutable key-value store for ordered data with a minimal memory footprint.
Data is stored in a crate::freezer::Freezer and a crate::ordinal::Ordinal to enable lookups by both index and key with minimal memory overhead.
§Uniqueness
Archive assumes all stored indices are unique. Writing to an occupied index is a no-op. If the same key is associated with multiple indices, there is no guarantee which value will be returned.
§Compression
Archive supports compressing data before storing it on disk. This can be enabled by setting
the compression field in the Config struct to a valid zstd compression level. This setting
can be changed between initializations of Archive, however, it must remain populated if any
data was written with compression enabled.
§Durability and Recovery
put updates the underlying crate::freezer::Freezer and crate::ordinal::Ordinal
eagerly, but data is not committed until sync succeeds. Sync first makes the freezer
and ordinal data durable, then commits metadata that names the freezer checkpoint and ordinal
section bits. On restart, this metadata is the source of truth: lower-layer data not described by
metadata is treated as uncommitted and may be removed during initialization. If no freezer
checkpoint has been committed yet, initialization starts from an empty archive.
§Querying for Gaps
Archive tracks gaps in the index space to enable the caller to efficiently fetch unknown keys
using next_gap. This is a very common pattern when syncing blocks in a blockchain.
§Example
use commonware_runtime::{Spawner, Runner, deterministic, buffer::paged::CacheRef};
use commonware_cryptography::{Hasher as _, Sha256};
use commonware_storage::{
archive::{
Archive as _,
immutable::{Archive, Config},
},
};
use commonware_utils::{NZUsize, NZU16, NZU64};
let executor = deterministic::Runner::default();
executor.start(|context| async move {
// Create an archive
let cfg = Config {
metadata_partition: "metadata".into(),
freezer_table_partition: "freezer-table".into(),
freezer_table_initial_size: 65_536,
freezer_table_resize_frequency: 4,
freezer_table_resize_chunk_size: 16_384,
freezer_key_partition: "freezer-key".into(),
freezer_key_page_cache: CacheRef::from_pooler(&context, NZU16!(1024), NZUsize!(10)),
freezer_value_partition: "freezer-value".into(),
freezer_value_target_size: 1024,
freezer_value_compression: Some(3),
ordinal_partition: "ordinal".into(),
items_per_section: NZU64!(1024),
freezer_key_write_buffer: NZUsize!(1024),
freezer_value_write_buffer: NZUsize!(1024),
ordinal_write_buffer: NZUsize!(1024),
replay_buffer: NZUsize!(1024),
codec_config: (),
};
let mut archive = Archive::init(context, cfg).await.unwrap();
// Put a key
archive.put(1, Sha256::hash(b"data"), 10).await.unwrap();
// Sync the archive
archive.sync().await.unwrap();
});