Crate ic_memory 

canic  ·  icydb  ·  ic-timers  ·  ic-memory  ·  ic-query  ·  ic-backup  ·  ic-blob-storage  ·  ic-testkit

Jump to: What it does · Why it matters · Is it useful? · How it works · Choose an integration style · Quick start · Troubleshooting · Advanced integration

What it does

ic-memory is a safety system for an Internet Computer application’s persistent data.

When an application is upgraded, its code changes but its stored data remains. Each database, log, queue, or settings store must reconnect to the same storage location it used before. If two stores are accidentally swapped, the new code can interpret one kind of data as another.

ic-memory remembers which storage location belongs to each named store. Before the application opens any of those stores, it checks the new layout against the saved ownership records. A conflicting upgrade stops with an error instead of opening the wrong data.

ic-memory protects the connection between a store and its storage location. It is not a backup system, a database schema migrator, or a data validator.

Why this matters

Imagine that version 1 of an application stores users and orders separately:

Users  -> storage location 100
Orders -> storage location 101

A later version accidentally reverses those locations:

Users  -> storage location 101
Orders -> storage location 100

The diagram below shows the same mistake and the point where ic-memory intervenes:

A changed store-to-location mapping is rejected before application data opens.

The program can still compile, and the upgrade can still install. Without an allocation check, it may then read order records as users and user records as orders. ic-memory detects the changed ownership before either location is opened.

This protection matters because a stable-memory location is a durable part of an application’s storage layout, even though it can look like an ordinary number in source code.

Is it useful for my application?

Use ic-memory for evolving multi-store layouts, not as a backup or migration system.

ic-memory is most useful for frameworks, generated canisters, multi-store applications, plugin systems, and canister families that evolve over time.

How it works

The lifecycle has three parts:

1. Name each store. The application gives every persistent store a permanent identity, such as app.users.v1.
2. Remember its location. On the first successful installation, ic-memory records which storage location belongs to each name.
3. Check before opening. On every installation or upgrade, the new application version declares the layout it expects. ic-memory compares that layout with the remembered one before any application store opens.

Name stores, retain their locations, and check the complete layout before opening data.

If the layouts agree, the application receives permission to open its stores. If a known name moved, a location changed owner, or another allocation rule is broken, the check returns an error and ic-memory grants no permission to open application stores.

This check-and-record operation is called bootstrap. Its important guarantee is validation before open: a diagnostic report or an uncommitted validation result cannot grant access to a store. The runtime publishes permission only after the allocation ledger has been recovered, checked, and durably updated.

The ledger stores one current ownership record per allocated identity. Store schema versions and schema admission belong to the application. Omitted identities retain their IDs to prevent accidental reuse. A commit counter and one checksummed current snapshot support stale-proof checks and corruption detection.

What happens when a check fails?

Bootstrap returns an error before application stores are opened. ic-memory does not silently repair, move, discard, or reinterpret a conflicting allocation. A failed attempt publishes no permission to open stores.

Treat the error as an upgrade-safety signal: keep the existing stable memory, inspect the requested keys, host namespace grants, exclusions, policy and diagnostics, then correct the new application version. Do not erase the allocation ledger or replace it with an empty one to make the error disappear. See the symptom-based troubleshooting guide for safe next steps.

What it protects

Allocation safety complements backups, schema migration, authorization, and disaster recovery.

Removing a store does not make its old location reusable: its key-to-ID binding is permanent. Application retirement and any rule forbidding the same key from reopening belong to the host/storage owner.

Terminology

• A stable key is a permanent store name, such as app.orders.v1.
• A memory ID is the physical MemoryManager location behind that key.
• An authority names the linked component requesting a key.
• A namespace grant admits that authority’s keys under a host-selected prefix.
• The allocation pool contains eligible application IDs shared by all owners.
• Bootstrap recovers, admits, resolves, validates and commits before opening.
• The allocation ledger retains permanent key-to-ID bindings, including omitted stores.

The application owns one pool and one bootstrap per backing memory. Libraries contribute key requests, verify that the host included them, and open their committed keys. Owner labels are current host policy; they are not persisted ownership history, caller authentication or a sandbox for linked code.

Quick start for developers

Add the crate:

[dependencies]
ic-memory = "0.46.0"

ic-memory exports Memory, DefaultMemoryImpl and VectorMemory for host backing setup. It does not re-export raw collections or memory managers. Production collection integrations belong in IC Storage’s governed facade, using the RuntimeMemory leases obtained from the host’s runtime. Memory’s internal manager, ledger and qualification fixtures retain their substrate use.

Declare keys; let the host allocate

Components name their owner and permanent keys without choosing IDs:

const MEMORY_AUTHORITY: &str = "example_app";
const USERS_KEY: &str = "example_app.users.v1";

fn component_requests() -> Result<Vec<ic_memory::MemoryRequest>, ic_memory::MemoryDeclarationError> {
    Ok(vec![ic_memory::MemoryRequest::new(
        MEMORY_AUTHORITY, USERS_KEY)?])
}

fn initialize_stable_storage() -> Result<(), Box<dyn std::error::Error>> {
    let pool = ic_memory::MemoryAllocationPool::new(
        vec![ic_memory::MemoryAuthority::new(MEMORY_AUTHORITY, "example_app.")?],
        vec![],
    )?;
    let declarations = ic_memory::SealedDeclarationSnapshot::new(component_requests()?)?;
    let mut runtime = ic_memory::MemoryRuntime::new(
        ic_memory::DefaultMemoryImpl::default(),
    )?;
    runtime.bootstrap(&declarations, &pool, &ic_memory::GenericAllocationPolicy)?;
    let users = runtime.open_memory(USERS_KEY)?;
    drop(users);
    Ok(())
}

The application retains this runtime across calls, bootstraps it during initialization and post-upgrade, and injects opened handles into components. Use MemoryRuntime::new_with_config for custom bucket geometry and pass the host policy to bootstrap. The library owns no singleton or thread-local runtime. Request factories only construct metadata; they never open stores.

Each allocation has at most one live lease per runtime. A second open_memory returns RuntimeOpenError::AlreadyOpen { id }; RuntimeMemory does not implement Clone. Transfer the handle into its collection and drop that owner before reopening. A collection’s memory-transfer operation keeps the same lease alive; failed construction and ordinary unwinding release it when the handle is dropped. For a log, obtain two different allocations for index and data memory.

The host must retain exclusive custody of the backing: do not create another runtime/manager or perform raw IO against aliases while the runtime or any of its handles lives. A handle can outlive its runtime, so dropping the runtime alone does not permit a replacement. Generic backing aliases cannot be detected by this crate. These are ownership rules for cooperating integrations, not an isolation boundary against arbitrary linked code. Never leak a handle to force reopening; a forgotten lease remains occupied.

Known keys retain their committed IDs. New requests are sorted by key and take the lowest unclaimed ID in the common pool. All retained records, including omitted keys, occupy their IDs permanently. Namespace grants do not divide space between components. Exhaustion rejects the entire attempt.

See examples/key_only.rs and examples/composed_host.rs.

Applications composed from several libraries

Each selected component supplies its complete request list. The host combines those lists into one SealedDeclarationSnapshot, admits disjoint namespaces for their named owners and supplies explicit physical exclusions for unmanaged MemoryManager clients. All components draw from the same remaining pool. The application bootstraps the combined layout once.

Libraries adopting an already bootstrapped host can verify that all of their requirements were included without rerunning bootstrap or replacing the host’s policy:

fn adopt_host<M: ic_memory::Memory>(
    runtime: &ic_memory::MemoryRuntime<M>,
) -> Result<(), Box<dyn std::error::Error>> {
    let requests = vec![ic_memory::MemoryRequest::new(
        "library_name", "library_name.rows.v1")?];
    let requirements = ic_memory::SealedDeclarationSnapshot::new(requests)?;
    runtime.verify_authority(
        &requirements,
        "library_name",
    )?;
    Ok(())
}

Duplicate keys, overlapping namespace grants, foreign owner claims and excluded historical IDs fail before application stores open. Populated IDs without a ledger record must be explicitly excluded by the host; bootstrap refuses to infer custody from stored bytes.

The runtime reserves IDs 0..=9 and keys under ic_memory.* for its own governance records. ID 0 contains the allocation ledger. Application code may use IDs through 254; ID 255 is the upstream unallocated sentinel and can never be declared.

Stable keys

A stable key names the durable identity of a store, not its current numeric location:

namespace.component.store_or_role.vN

use ic_memory::StableKey;

StableKey::parse("app.orders.v1").expect("app key");
StableKey::parse("myapp.audit_log.v1").expect("app key");
StableKey::parse("icydb.test_db.users.data.v1").expect("database key");

Changing a key creates a new allocation identity. If the durable store is still the same store, keep its key and apply schema changes through the storage owner.

Frequently asked questions

Does ic-memory move or migrate application data?

No. It validates allocation identity. Schema and data migrations remain the application’s responsibility.

Does it back up stable memory or recover corrupted application data?

No. Keep a separate backup and disaster-recovery plan. ic-memory fails closed when its allocation metadata cannot be recovered safely.

What happens when validation fails?

Bootstrap returns an error and publishes no open capability. Fix the proposed layout or policy; do not erase the ledger to bypass the conflict.

Can a retired memory location be reused?

No. Its permanent key-to-ID binding prevents another key from opening unrelated data at that location. The host/storage owner decides whether the same key may reopen a logically retired store.

Does every library bootstrap separately?

No. One owner bootstraps each concrete runtime. Libraries verify their requirements against the host’s committed layout and open only their keys.

What does each library declare?

Each library declares permanent keys and its owner label. The host admits the namespace and owns physical exclusions; Memory assigns IDs and retains them.

Can an upgrade add a new store safely?

Yes, provided its key is new, the host admits its namespace and owner, the common pool has space, and the complete layout passes admission and validation.

Operations and advanced integration

Custom policies return PolicyIdentity directly from runtime_bootstrap_identity. Construct it with PolicyIdentity::new(host_token), where host_token is one [u8; 32] identifying the complete prepared policy configuration and semantics. Prepare any fallible host configuration first. Hosts own descriptive labels, versions and token derivation; change the token when that policy changes. Memory compares all token bytes for warm binding and does not persist or hash them. Diagnostic JSON returns token arrays; fixture doctor reports include the tested token.

Production recovery and inspection use the host runtime’s diagnostic_export, commit_recovery_diagnostic, memory_allocations and memory_allocation_summary. Hosts own operator response assembly from these bounded queries and typed errors. The combined testing::doctor_report(&runtime, &declarations, &pool, &policy) preflight helper and its report types require test-support. It is read-only and never runs recovered-admission preparation or grants open authority. Raw ledger DTOs, bounded raw commit-store readers and passive export construction also require test-support; recovered authority remains internal. Bounded Cell payload inspection remains available without granting allocation authority.

The current durable ledger uses checked numeric slots and the ICMKEY01 checksummed snapshot. Its protected header owns the generation; the bounded CBOR byte-string payload contains only the allocation-record array. Recovery constructs the logical ledger with that checked generation. The 0.48.0 format cut retires earlier images, including ICMSNAP1 and pending ICMREC01: use fresh/reset memory. Invalid or incompatible bytes reject without initializing genesis or clearing memory. See the current ledger contract.

For fixture runtimes, opt in to test-support and call testing::fixture_runtime with explicit backing memory, owner, namespace, stable-key list and bucket configuration. It returns MemoryRuntime for opening multiple allocations or exercising refused opens, without initializing collections. Setup uses the ordinary generic allocation policy and has typed FixtureRuntimeError failures; no global manager or production admission policy is installed.

For fixture-only byte cells, opt in to test-support and construct testing::FixtureCell with explicit backing memory, owner, namespace, key and MemoryManagerConfig. The Cell handle retains its backing; the temporary bootstrap runtime is dropped. No lifecycle handler or singleton is installed. Use create_bounded or open_bounded for explicit fresh/retained admission with a caller-selected byte limit. Setup failures retain their typed category under FixtureCellError::Setup(FixtureRuntimeError). Retained headers are checked before payload loading; malformed cells are refused without initialization. try_set refuses oversized writes before mutation; into_memory transfers the admitted handle for caller-owned raw inspection. Callers keep codecs, work budgets and restored-state/lifecycle validation; fresh empty bytes must never substitute for a required retained journal. Bootstrap may update ledger metadata before a later Cell refusal, and backing IO can still panic.

For frozen sparse backing fixtures, testing::memory_from_sparse_hex expands ordered (offset, hex) runs into fresh zero-filled VectorMemory with an explicit page ceiling. It checks bounds and hex before backing allocation and reports typed failures. Callers retain provenance, schema and payload validation. This helper does not open a runtime, recover authority or convert retained formats.

Production requests, namespace grants and allocation pools use checked Rust constructors. Their serialization remains available for diagnostic output; Deserialize requires test-support for explicit fixtures. Applications that accept external configuration own their input DTOs and pass decoded fields to Memory’s constructors. This does not change durable ledger decoding.

Resolved numeric declarations are read-only runtime outputs. Production callers submit MemoryRequest values and borrow runtime.committed_allocations() after bootstrap; the returned generation is also available through committed_generation(). Explicit numeric declaration builders and decoding require test-support; their construction errors live in testing. Per-ID physical reports contain numeric facts only. Hosts project current labels and eligibility from the runtime’s borrowed bindings and allocation_pool(); see operations guidance.

• Operations and diagnostics explains memory attribution, bucket configuration, and runtime reports.
• Troubleshooting maps common symptoms to safe recovery steps.
• Advanced ic-memory covers explicit composition, custom policies, recovery, and runtime-owned persistence.
• Safety invariants records the guarantees future changes must preserve.
• Documentation index separates current guidance from historical engineering evidence.
• Release guide is for maintainers preparing and publishing a release.

ic-memory is pre-1.0 infrastructure extracted from Canic. Current releases use the current API and durable format only; earlier pre-1.0 wire formats are not a compatibility target. Stable-memory allocation-governance primitives for Internet Computer canister upgrades.

ic-memory prevents stable-memory slot drift.

Once a stable key is committed to a physical allocation slot, future binaries must either reopen that same stable key on that same slot or declare a new stable key.

The crate records and validates durable ownership in both directions: an active stable key cannot move to a different physical slot, and an active physical slot cannot be reused by a different stable key.

The intended runtime integration flow is:

1. Recover the persisted allocation ledger.
2. Admit consumer identity and authorized historical selections from bounded metadata under the host’s RuntimeBootstrapPolicy.
3. Resolve key requests in the host-wide pool under current namespace grants, then validate retained key-to-ID bindings and application policy.
4. Stage and durably persist the next generation.
5. Only then open stable-memory handles through committed allocation authority.

This crate owns allocation invariants, not framework policy. Namespace rules, controller authorization, endpoint lifecycle, schema migrations, and application validation belong to the framework or application.

The host supplies one MemoryAllocationPool with explicit namespace grants and physical exclusions. Linked components request permanent keys and name their owner; they do not select numeric IDs or reserve component subranges. Namespace grants are current host policy, not caller authentication or persisted ownership labels. Application admission remains host-owned.

Use these primitives before opening stable-memory handles. Integrations should recover the historical ledger, declare the stores expected by the current binary, admit recovered identity, resolve requests, validate against history and policy, persist a new generation, and only then publish authority before opening slots through the storage owner.

Bounded physical attribution is available through MemoryRuntime::memory_allocations and MemoryRuntime::memory_allocation_summary. It reports actual persisted buckets and explicit residuals without decoding retained ownership. Virtual extent is not payload occupancy. Opens return RuntimeMemory; explicit MemoryManagerConfig selects fresh-state buckets or checks a persisted setting without migration. The default remains 128 pages.

MemoryRuntime is the canonical owner for one backing memory instance. It owns that memory’s manager, ledger persistence, bootstrap lifecycle, committed bindings, opens, and diagnostics. Each bootstrap attempt fallibly decodes its ledger record and uses a temporary cell for capacity-checked writes. Linked code contributes declarations to one immutable SealedDeclarationSnapshot, which is supplied to each runtime independently.

The host supplies the complete checked request snapshot, allocation pool and admission policy directly to the runtime. Component factories and generators contribute request lists; opening a store never registers a declaration. Exactly one runtime owns bootstrap and persistence for each backing memory.

ic-stable-structures MemoryManager IDs are the first-class supported physical slot substrate. That ID domain is u8: IDs 0..=254 are usable, and ID 255 is always the ic-stable-structures unallocated sentinel. Protected current-snapshot commits and stable-cell encoding remain internal to the runtime. Public diagnostics describe their state without granting a manual commit or persistence-confirmation route.

Hosts use DefaultMemoryImpl, VectorMemory and Memory for backing setup. RuntimeMemory is an exclusive allocation lease; production typed collections consume these leases through IC Storage’s governed wrappers.

Structs

AllocationDeclaration

AllocationDeclaration

AllocationRecord

AllocationRecord

BootstrapAdmission

BootstrapAdmission

DiagnosticExport

DiagnosticExport

DiagnosticMemorySize

DiagnosticMemorySize

DiagnosticRecord

DiagnosticRecord

GenericAllocationPolicy

GenericAllocationPolicy

MemoryAllocation

MemoryAllocation

MemoryAllocationPool

MemoryAllocationPool

MemoryAllocationSummary

MemoryAllocationSummary

MemoryAllocations

MemoryAllocations

MemoryAuthority

MemoryAuthority

MemoryBindingSummary

MemoryBindingSummary

MemoryManagerConfig

MemoryManagerConfig

MemoryManagerSlot

MemoryManagerSlot

MemoryRequest

MemoryRequest

MemoryRuntime

MemoryRuntime

PolicyIdentity

PolicyIdentity

RuntimeMemory

RuntimeMemory

SealedDeclarationSnapshot

SealedDeclarationSnapshot

StableKey

StableKey

StableKeyError

StableKeyError

Enums

AllocationStageError

AllocationStageError

AllocationValidationError

AllocationValidationError

BootstrapAdmissionError

BootstrapAdmissionError

CommitRecoveryError

CommitRecoveryError

CommitSnapshotDiagnostic

CommitSnapshotDiagnostic

LedgerCommitError

LedgerCommitError

LedgerIntegrityError

LedgerIntegrityError

MemoryAllocationPoolError

MemoryAllocationPoolError

MemoryDeclarationError

MemoryDeclarationError

MemoryManagerLayoutError

MemoryManagerLayoutError

MemoryManagerSlotError

MemoryManagerSlotError

MemoryResolutionError

MemoryResolutionError

RuntimeAdoptionError

RuntimeAdoptionError

RuntimeBootstrapError

RuntimeBootstrapError

RuntimeConstructionError

RuntimeConstructionError

RuntimeDiagnosticError

RuntimeDiagnosticError

RuntimeGrowError

RuntimeGrowError

RuntimeOpenError

RuntimeOpenError

StableCellLedgerError

StableCellLedgerError

StableCellPayloadError

StableCellPayloadError

Constants

IC_MEMORY_AUTHORITY_OWNER

Diagnostic owner label for ic-memory allocation-governance infrastructure.

IC_MEMORY_LEDGER_STABLE_KEY

Stable key of the allocation ledger when backed by the current MemoryManager substrate.

IC_MEMORY_STABLE_KEY_PREFIX

Stable-key namespace prefix reserved for ic-memory allocation-governance infrastructure.

MAX_LEDGER_BYTES

Maximum encoded ownership ledger size (64 KiB for at most 255 records).

MAX_LEDGER_NESTING

Maximum CBOR container nesting on maintained decode paths.

MAX_LEDGER_RECORD_BYTES

One bounded CBOR byte string plus record metadata (64 KiB + 4 KiB).

MEMORY_MANAGER_GOVERNANCE_MAX_ID

Last MemoryManager ID reserved for ic-memory governance in the current substrate.

MEMORY_MANAGER_INVALID_ID

MemoryManager unallocated-bucket sentinel. This is not a usable slot.

MEMORY_MANAGER_LEDGER_ID

MemoryManager ID used by the allocation ledger in the current MemoryManager substrate.

MEMORY_MANAGER_MAX_ID

Last usable MemoryManager virtual memory ID.

MEMORY_MANAGER_MIN_ID

First usable MemoryManager virtual memory ID.

STABLE_CELL_HEADER_SIZE

Stable-cell header byte length.

STABLE_CELL_LAYOUT_VERSION

Stable-cell layout version supported by this adapter.

STABLE_CELL_MAGIC

Stable-cell magic prefix written by ic-stable-structures::Cell.

STABLE_CELL_VALUE_OFFSET

Byte offset where the stable-cell value payload starts.

WASM_PAGE_SIZE_BYTES

WebAssembly page size used by ic-stable-structures memory implementations.

Traits

Memory

Backing setup and IO interfaces from the exact substrate dependency. Hosts own backing custody; production collections belong behind IC Storage’s governed wrappers and consume RuntimeMemory leases. Import visibility is not admission: implementing Memory does not construct a checked handle. Collection codec traits belong to Storage; metric records and restoration belong to IC Metrics Stable. Abstraction over a WebAssembly-style linear memory (e.g., stable memory).

RuntimeBootstrapPolicy

RuntimeBootstrapPolicy

Functions

decode_stable_cell_payload

Decode the raw value payload from an ic-stable-structures::Cell memory.

is_ic_memory_stable_key

Return true when stable_key belongs to the ic-memory namespace.

validate_memory_manager_id

Validate that a MemoryManager ID is usable as an allocation slot.

Type Aliases

DefaultMemoryImpl

Backing setup and IO interfaces from the exact substrate dependency. Hosts own backing custody; production collections belong behind IC Storage’s governed wrappers and consume RuntimeMemory leases. Import visibility is not admission: implementing Memory does not construct a checked handle. Collection codec traits belong to Storage; metric records and restoration belong to IC Metrics Stable.

VectorMemory

Backing setup and IO interfaces from the exact substrate dependency. Hosts own backing custody; production collections belong behind IC Storage’s governed wrappers and consume RuntimeMemory leases. Import visibility is not admission: implementing Memory does not construct a checked handle. Collection codec traits belong to Storage; metric records and restoration belong to IC Metrics Stable. A Memory that is based on a vector.