ic-memory

---

ic-memory helps Internet Computer canisters avoid opening the wrong stable memory after an upgrade.

It remembers this mapping forever:

logical store -> physical stable-memory slot

If a future version tries to move that store to a different slot, or reuse that slot for a different store, ic-memory rejects the layout before stable-memory handles are opened.

Key-only allocation

Applications can request durable keys while the host grants an explicit pool:

ic_memory::ic_memory_range!(authority = "app", start = 10, end = 254, mode = Allowed);
ic_memory::ic_memory_declaration!(authority = "app", key = "app.users.v1");

fn initialize() {
    ic_memory::bootstrap_default_memory_manager().unwrap();
    let assigned_id = ic_memory::default_memory_manager_memory_id("app.users.v1").unwrap();
    let users = ic_memory::open_default_memory_manager_memory_by_key("app.users.v1").unwrap();
    # let _ = (assigned_id, users);
}

Known keys retain their committed IDs. New requests are sorted by stable key and take the lowest unclaimed ID in their authority's explicit Allowed grants. Governance IDs, fixed claims, reservations, omitted allocations and retired allocations remain unavailable. Reserved ranges do not supply new automatic slots. A matching historical reservation activates through normal claim validation and current policy.

ic_memory_range! defaults to Reserved when mode is omitted. Always pass mode = Allowed for a pool that must serve new key-only requests. Without a free ID in a matching Allowed range, resolution returns MemoryResolutionError::Exhausted, even if a reserved range has free IDs.

A composed host grants each library only its intended ranges and bootstraps once. Libraries resolve IDs with default_memory_manager_memory_id(...) and open by key without changing the host's policy or bucket profile. Hosts must declare or reserve allocations used by raw MemoryManager clients before admitting automatic requests; physical diagnostics cannot infer their ownership.

Libraries can verify their requirements against the host's current commitment:

fn adopt_host() -> Result<(), Box<dyn std::error::Error>> {
    let requirements = ic_memory::sealed_declaration_snapshot()?;
    ic_memory::verify_default_memory_manager_authority(&requirements, "icydb.example")?;
    Ok(())
}

For an owned runtime, use runtime.verify_authority(&requirements, authority) and runtime.memory_id(key). Verification requires at least one fixed declaration or logical request under that authority and checks every matching requirement. Fixed declarations must match key, ID, label and diagnostic schema. Logical requests must match key, authority and diagnostic schema while adopting the host's assigned ID. Other authorities and additional committed keys are ignored. Typed errors identify missing keys, wrong IDs, authority mismatches and metadata mismatches. These helpers never bootstrap, replay admission or change policy, grants or bucket configuration. Application schema and lifecycle checks remain with the consumer.

CommittedAllocations contains immutable declarations with validated metadata and unique keys and slots. Consumers may rely on those guarantees without rebuilding uniqueness sets; they still validate live store and journal contents.

Hosts can also implement RuntimeBootstrapPolicy::prepare_bootstrap to inspect validated recovered allocation metadata and explicitly include known historical journals before the single commit. The context grants no memory access; unknown, retired or unauthorized selections reject. Warm adoption does not rerun admission.

See the runnable standalone and composed example, composed-host cold-reopen and native-thread example, recovered-journal example, admission contract, and recovery limits and omitted-store inspection.

In native tests the default runtime and backing are thread-local. Each worker must bootstrap the host with its selected policy and bucket size before library adoption or initialization of thread-local stable stores. A database's readiness does not bootstrap other host stores. Sharing a package identity is necessary for composition, but also qualify admission, grants and retained data across repeated cold reopens of the same release and backing.

Why Use It?

Use ic-memory when a canister has more than one stable store and the layout can change over time.

It is most useful for frameworks, generated canisters, multi-store apps, plugin systems, and canister families that evolve across releases.

You probably do not need it for a tiny canister with one hand-written stable structure and a fixed layout.

The Bug

Version 1 ships with:

app.users.v1  -> MemoryManager ID 100
app.orders.v1 -> MemoryManager ID 101

A later upgrade accidentally ships with:

app.users.v1  -> MemoryManager ID 101
app.orders.v1 -> MemoryManager ID 100

That can still compile. It can even install.

But now the canister may open orders data as users data, and users data as orders data. ic-memory catches that mismatch first.

Quick Start

Declare the dependency:

[dependencies]
ic-memory = "0.23.0"

ic-memory re-exports its exact ic-stable-structures dependency through ic_memory::ic_stable_structures. Import collections, backing memories, and traits through that namespace to use the same upstream types as the runtime:

use ic_memory::{
    RuntimeMemory,
    ic_stable_structures::{Cell, DefaultMemoryImpl},
};

type CounterStore = Cell<u64, RuntimeMemory<DefaultMemoryImpl>>;

Memory, Storable, storable::Bound, and the other upstream collections are available through the same namespace. A separate ic-stable-structures dependency is unnecessary for these imports. Initialize stores with handles opened by MemoryRuntime or the default runtime, which owns the canister's memory manager.

Declare the MemoryManager IDs your crate owns. A shared compile-time constant keeps the explicit authority identical across the range and each key:

const MEMORY_AUTHORITY: &str = "icydb.test_db";

ic_memory::ic_memory_range!(authority = MEMORY_AUTHORITY, start = 120, end = 129);

This fixed-ID example uses the default Reserved mode. For new key-only requests, declare the host's pool with mode = Allowed, as shown above.

The authority string is explicit stable policy metadata. It is not persisted allocation identity; the stable key and memory ID fill that role. Use the same authority value for the package's range and key declarations, and do not derive it from a Cargo package name or module path.

Open stable structures through ic_memory_key!:

use std::cell::RefCell;

thread_local! {
    pub static USERS: RefCell<UsersStore> = RefCell::new(UsersStore::init(
        ic_memory::ic_memory_key!(
            authority = MEMORY_AUTHORITY,
            key = "icydb.test_db.users.data.v1",
            ty = UsersStore,
            id = 120,
        )
        .expect("committed users memory")
    ));
}

Bootstrap once per concrete memory runtime before touching stable data:

#[ic_cdk::init]
fn init() {
    ic_memory::bootstrap_default_memory_manager().expect("valid stable-memory layout");
}

#[ic_cdk::post_upgrade]
fn post_upgrade() {
    ic_memory::bootstrap_default_memory_manager().expect("valid stable-memory layout");
}

That is the normal path.

The default runtime API is exported from the crate root. It is one thread-local MemoryRuntime<DefaultMemoryImpl>, so every native thread owns an independent backing memory, lifecycle, committed capability, and diagnostic view. On IC Wasm, execution is single-threaded and the same TLS object naturally has canister-instance lifetime.

Use helpers such as ic_memory::bootstrap_default_memory_manager(), ic_memory::bootstrap_default_memory_manager_with_policy(...), ic_memory::committed_allocations(), ic_memory::open_default_memory_manager_memory(...), and the macros shown above; implementation modules are private.

The no-argument bootstrap helper uses GenericRangePolicy, ic-memory's built-in policy with its existing versioned PolicyIdentity. The runtime enforces range ownership and internal reservations; this policy adds no application restrictions. A custom policy implements both AllocationPolicy and RuntimeBootstrapPolicy. Its bounded identity contains a policy-family name, a nonzero semantic version, and an optional caller-computed 32-byte configuration digest. Change the version when policy semantics change and use the digest when effective runtime configuration changes.

Multi-Crate Composition

Every crate registers into the same linked declaration registry. Crates do not need to import or name each other:

mod package_a {
    ic_memory::ic_memory_range!(authority = "package_a", start = 100, end = 109);

    thread_local! {
        pub static USERS: RefCell<UsersStore> = RefCell::new(UsersStore::init(
            ic_memory::ic_memory_key!(
                authority = "package_a",
                key = "package_a.users.v1",
                ty = UsersStore,
                id = 100,
            )
            .expect("committed users memory")
        ));
    }
}

mod package_b {
    ic_memory::ic_memory_range!(authority = "package_b", start = 110, end = 119);

    thread_local! {
        pub static ORDERS: RefCell<OrdersStore> = RefCell::new(OrdersStore::init(
            ic_memory::ic_memory_key!(
                authority = "package_b",
                key = "package_b.orders.v1",
                ty = OrdersStore,
                id = 110,
            )
            .expect("committed orders memory")
        ));
    }
}

The linked program seals one immutable, canonical declaration snapshot. Bootstrap supplies that snapshot to the calling thread's default runtime, recovers and commits that runtime's allocation ledger, and publishes committed allocations into that runtime only. TLS-backed stores open when your code first touches the thread_local!.

Duplicate stable keys, duplicate MemoryManager IDs, overlapping ranges, and out-of-range declarations fail before stable structures open.

Logical requests require explicit Allowed grants for fresh assignments, even when the host supplies a custom policy. A Reserved range supports fixed or matching historical claims but supplies no fresh logical placement pool.

ic-memory follows the ic-stable-structures::MemoryManager ID domain exactly: IDs 0..=254 are usable, and ID 255 is always the unallocated sentinel. It is not an application slot and cannot be declared or reserved.

The default runtime reserves MemoryManager IDs 0..=9 and stable keys under ic_memory.* for allocation-governance records. The ledger itself lives at ID 0; it remains in the durable ledger for recovery, but public runtime helpers do not publish or open that internal allocation as application memory.

Range claims are authoritative in the default runtime. If a crate registers ic_memory_range!, its declared memories must stay inside that range. Framework adapters register the ranges they want ic-memory to enforce. When no user ranges are registered, fixed claims can use the adapter's own application policy. Logical placement and historical selection always require explicit grants.

The committed allocation state is an in-memory capability published into one runtime only after that runtime's stable-cell persistence succeeds. It is not a serde payload and should not be treated as configuration.

Explicit Runtimes

Frameworks and tests that own backing memory directly should use MemoryRuntime<M> as the canonical API:

use ic_memory::{MemoryRuntime, sealed_declaration_snapshot};

let declarations = sealed_declaration_snapshot()?;
let mut runtime = MemoryRuntime::new(backing_memory)?;
runtime.bootstrap(&declarations, &policy)?;

let rows = runtime.open_memory("app.rows.v1", 120)?;
let diagnostics = runtime.diagnostic_export()?;

Construction is fallible. Empty backing memory is initialized as an ic-stable-structures MemoryManager; nonempty backing memory must already pass validation of the current MGR header, bucket table, and virtual/physical extents. Foreign, unsupported, or corrupt metadata returns a typed error before manager initialization can write. The read-only layout adapter is coupled to the exact ic-stable-structures = "=0.7.2" dependency. A pre-grown blank memory is nonempty and is rejected rather than assumed disposable.

Each runtime owns all facts derived from backing_memory: recovery, ledger cell, lifecycle, committed allocations, opens, diagnostics, and live sizes. Multiple runtimes share only the immutable linked declaration snapshot. A failed bootstrap publishes no capability, and repeated bootstrap on the same runtime object is idempotent only when the snapshot and RuntimeBootstrapPolicy::runtime_bootstrap_identity() match the successful bootstrap. Independently sealed snapshots with equal canonical contents also match. A changed snapshot or policy identity returns a typed error without touching the ledger. Policy implementations should change their identity whenever policy configuration or semantics change. This binding is intentionally in-memory lifecycle and diagnostic state; it is not upgrade audit history and is not persisted in the allocation ledger.

There is intentionally no public reset API. Native tests should construct a new explicit runtime or use the naturally independent default TLS runtime; changing global flags cannot reset a concrete stable-memory instance safely.

Bounded physical allocation attribution

Use runtime.memory_allocations() or default_memory_manager_memory_allocations() for an owned MemoryAllocations report. Collection reads exactly 34,848 bytes of validated manager metadata and returns all 255 usable IDs in order, including zero-size IDs and the ledger at ID 0. It never decodes ledger history, initializes stores, writes, grows memory, or advances a generation. The default helper refuses to construct a missing runtime; an existing unbootstrapped runtime can report physical allocation with unknown current bindings.

For recurring numeric metrics, use runtime.memory_allocation_summary() or default_memory_manager_memory_allocation_summary(). The owned, copyable MemoryAllocationSummary returns the same totals and capacity facts plus allocated bytes and bucket slack for current_binding, ledger_binding and unknown_binding. It checks all 255 IDs with the same 34,848-byte metadata read, without constructing per-ID rows or copying keys, owners or range claims. It does not decode history or measure payload occupancy. Both reports share the same validated measurement and accounting path; the detailed report remains available for protected inspection. The default summary helper does not construct a missing runtime.

The report measures the actual persisted bucket size, physical and virtual extents, assigned buckets, manager metadata, known current stable-key/owner bindings, and unknown/unmanaged residuals. Virtual bytes are addressable extent, not payload occupancy. payload_bytes is unavailable. Bucket slack is only assigned bucket capacity beyond virtual extent. Conservation is explicit:

physical bytes = manager metadata + assigned bucket bytes + unmanaged bytes
assigned bucket bytes = sum(per-ID bucket bytes)
                      = known binding bytes + unknown binding bytes
                      = virtual bytes + bucket slack

A current range claim does not prove historical ownership or grant access. Retired/absent keys are explicitly unknown; the ledger's reserved ID is included without reading its payload. Keep operator/controller authorization in the integrating application. Full doctor/ledger diagnostics below still decode history and are not substitutes for this bounded report.

Bucket policy

Fresh runtimes retain the 128-page (8 MiB) default. MemoryRuntime::new honors an existing same-release memory's actual setting. For an explicit setting use MemoryRuntime::new_with_config(memory, MemoryManagerConfig::new(pages)?); all nonzero u16 page counts are supported. Existing memory must match exactly or construction fails before effects. Configuration is immutable for that runtime.

For a default runtime, select configuration on the first bootstrap through bootstrap_default_memory_manager_with_config(config, &policy). Repeated explicit configuration must match the established manager, independently of the allocation policy identity. No bucket setting shrinks existing memory or migrates the durable format.

For configured bootstrap without a custom application policy, pass &ic_memory::GenericRangePolicy to the same helper. There is no second configured bootstrap path, profile state, or policy identity.

is_default_memory_manager_bootstrapped() and committed_allocations() do not construct a missing runtime. They return false / NotBootstrapped respectively, without initializing backing memory or choosing bucket size. Frameworks may inspect committed allocations first, adopt an already bootstrapped host runtime, and otherwise bootstrap with their chosen configuration. Adoption still requires checking the framework's declarations; do not re-bootstrap with the generic policy to bypass an existing host policy. Cached construction and TLS access failures remain errors, not absence. Default runtime construction belongs to bootstrap.

Default-runtime opens and default_memory_manager_memory_id() also leave a missing runtime untouched. An early open returns NotBootstrapped; it cannot silently select 128-page buckets and prevent a later configured bootstrap. Default export, commit-recovery and doctor diagnostics also leave a missing runtime untouched and return RuntimeDiagnosticError::NotBootstrapped. Early inspection therefore preserves the bootstrap owner's bucket configuration.

Open operations and macros return RuntimeMemory<M>, implementing Memory and Clone without requiring M: Clone. Stable store type annotations must use ic_memory::RuntimeMemory<DefaultMemoryImpl>. The runtime retains one private shared backing for read-only attribution and owns exactly one manager. Both safe and unsafe reads delegate to upstream, preserving specialized read_unsafe implementations without extra destination initialization in the runtime. Custom backings can continue using the Memory trait's default method.

RuntimeMemory::grow reserves physical backing capacity before assigning manager buckets and returns Result<u64, RuntimeGrowError> with the previous page count on success. Backing refusal, arithmetic overflow, reentry and bucket exhaustion return distinct typed errors without changing virtual extents or manager metadata. All handles, including clones and the ledger, share one live bucket count recovered from validated metadata at construction. Growth performs no metadata reads or table scans. The count is transient; the current durable format is unchanged. Native backing panics and partial writes remain outside this refusal guarantee. This is a hard cut for direct callers: replace integer checks with ?, match, or an explicit error handler. The ic_stable_structures::Memory trait requires an i64 result; only that adapter converts typed growth errors to -1. Collections retain their own behavior at that trait boundary. Ledger bootstrap propagates growth errors through RuntimeBootstrapError::LedgerGrowth.

The CANIC-162 handoff contains the exact Canic integration example, reproducible measurements, capacity tradeoffs, and limitations. Fixture evidence supports configurable smaller buckets, but does not justify changing the default or selecting a Toko policy without live attribution and a capacity assessment.

Diagnostics

Use default_memory_manager_doctor_report() for operator-facing preflight and runtime diagnostics. It returns NotBootstrapped if no default runtime exists, without initializing memory, choosing configuration or sealing declarations. TLS access and construction failures remain typed errors. An existing runtime can be inspected before or after bootstrap; the report includes the stable-cell status, protected commit recovery state, recovered ledger export, registered declarations, range authority, validation preflight, and live MemoryManager slot sizes when they can be recovered. This no-argument entry point evaluates the built-in policy. Integrations that bootstrap with a custom policy should call default_memory_manager_doctor_report_with_policy(&policy), or call runtime.doctor_report(&declarations, &policy) on an explicit runtime.

Doctor output includes the tested policy identity and sealed-declaration fingerprint, the binding established by successful bootstrap, and a typed binding comparison. Runtime size measurement uses validated recovered allocations; an invalid persisted slot rejects ledger recovery before any sizes are measured. Diagnostic failures carry stable DiagnosticCode values alongside their human-readable messages for operator automation. Doctor validation checks the supplied declarations and allocation policy; it does not run prepare_bootstrap or certify consumer admission.

Use default_memory_manager_commit_recovery_diagnostic() when you only need commit-slot presence and validity, the selected authoritative generation, and any corruption or ambiguity error. It also requires an existing runtime; default_memory_manager_diagnostic_export() additionally requires completed bootstrap. For prebootstrap inspection with explicit configuration, construct MemoryRuntime::new_with_config(memory, config) and call its recovery or doctor methods before bootstrap.

Stable Keys

Stable keys are permanent logical store names. They should describe ownership and purpose, not the current memory ID.

namespace.component.store_or_role.vN

Examples:

use ic_memory::StableKey;

StableKey::parse("app.orders.v1").expect("app key");
StableKey::parse("myapp.audit_log.v1").expect("app key");
StableKey::parse("icydb.test_db.users.data.v1").expect("database key");

Changing a key creates a new logical allocation identity. If the durable store is the same, keep the stable key and update schema metadata instead.

Schema metadata is optional diagnostic metadata for the in-place store schema. Construct it with SchemaMetadata::new(Some(version)); version 0 is reserved for absence and is rejected.

Releases

The release targets follow Canic's validate, bump, commit, tag, and push flow, adapted for this single library crate. They require Python 3.11+, Git, Make, Rust 1.99.0 with Clippy/rustfmt and wasm32-unknown-unknown, and the declared MSRV toolchain. Publishing also requires crates.io credentials configured for Cargo.

Commit the implementation and a nonempty, numbered entry at the top of CHANGELOG.md for the next version before starting.

The maintainer owns commits, tags and pushes, including commands that perform them. Agents leave source and documentation unstaged for review; see the repository instructions.

The maintainer can then use:

make release-patch   # Validate, bump patch, commit, annotate tag, push
make release-minor   # Validate, bump minor/reset patch, commit, annotate tag, push
make publish-dry-run # Verify the tagged release without uploading
make publish        # Publish the tagged release to crates.io

The release targets push the current branch and its vX.Y.Z tag atomically to origin. Publication is a separate command. PUBLISH_DRY_RUN=1 make publish also performs a dry run. Branches must already exist on origin, and the refreshed remote branch must be an ancestor of the local source commit.

make patch and make minor stop after validation and version preparation for local review. Finish with make release-stage, make release-commit, and make release-push, in that order. A rejected push can be retried with make release-push; do not bump the version again. A failed tag step can be retried with make release-commit without making another commit.

Preparation updates only Cargo.toml and the README dependency example, and refreshes the ignored local Cargo.lock. It verifies the final package and restores those files if preparation fails. Release commits must contain only the expected version edits and are bound to the validated source commit. Dirty trees, stale prepared state, unrelated staged changes, and conflicting release tags are rejected. The lockfile remains untracked.

make validate runs the release-flow regression tests, formatting, Clippy, serialized Rust tests and doctests, Wasm budget regression tests, Wasm checks and size budgets, the declared MSRV check, and package verification. CI shares its installed-toolchain checks with make validate-toolchain and reads the MSRV from Cargo.toml in a separate job. Development, CI, and the default VALIDATION_TOOLCHAIN read the Rust 1.99.0 pin from rust-toolchain.toml. The crate's declared MSRV remains Rust 1.88.0. make test-release-flow exercises the release commands in disposable repositories with a fake Cargo executable; it never publishes packages or contacts a hosted Git remote.

make test-wasm-size checks artifact discovery and budget failures using a fake Cargo executable without Git operations. make wasm-size resolves the actual artifact directory through Cargo metadata, including CARGO_TARGET_DIR and Cargo configuration, so stale files in a different target directory cannot satisfy the budgets.

More Detail

The short version:

declare ranges
register stable stores
seal linked declarations
bootstrap once per memory runtime
only then open stable memory

Framework authors and policy adapters should read ADVANCED.md. The non-negotiable invariants are recorded in SAFETY.md. The protocol whitepaper lives in whitepaper/src/SUMMARY.md and builds as an mdBook with make maintainer-build.

ic-memory is early infrastructure extracted from Canic. It owns allocation governance, not schema migration, endpoint routing, authorization, or data semantics.