Crate ic_memory

Expand description

IC Memory — Stops upgrades from mixing up stored data

canic · icydb · ic-timers · ic-memory · ic-query · ic-backup · ic-blob-storage · ic-testkit

Jump to: What it does · Why it matters · Is it useful? · How it works · Choose an integration style · Quick start · Troubleshooting · Advanced integration

§What it does

ic-memory is a safety system for an Internet Computer application’s persistent data.

When an application is upgraded, its code changes but its stored data remains. Each database, log, queue, or settings store must reconnect to the same storage location it used before. If two stores are accidentally swapped, the new code can interpret one kind of data as another.

ic-memory remembers which storage location belongs to each named store. Before the application opens any of those stores, it checks the new layout against the saved ownership records. A conflicting upgrade stops with an error instead of opening the wrong data.

ic-memory protects the connection between a store and its storage location. It is not a backup system, a database schema migrator, or a data validator.

§Why this matters

Imagine that version 1 of an application stores users and orders separately:

Users  -> storage location 100
Orders -> storage location 101

A later version accidentally reverses those locations:

Users  -> storage location 101
Orders -> storage location 100

The diagram below shows the same mistake and the point where ic-memory intervenes:

Before an upgrade, Users uses storage 100 and Orders uses storage 101. A mistaken upgrade swaps those assignments, so ic-memory blocks the upgrade before either store opens.

A changed store-to-location mapping is rejected before application data opens.

The program can still compile, and the upgrade can still install. Without an allocation check, it may then read order records as users and user records as orders. ic-memory detects the changed ownership before either location is opened.

This protection matters because a stable-memory location is a durable part of an application’s storage layout, even though it can look like an ordinary number in source code.

§Is it useful for my application?

Decision guide: ic-memory is most useful for applications with several persistent stores, libraries or plugins that contribute stores, or storage layouts that change across upgrades. It is not a backup or schema-migration tool.

Use ic-memory for evolving multi-store layouts, not as a backup or migration system.

ic-memory is most useful for frameworks, generated canisters, multi-store applications, plugin systems, and canister families that evolve over time.

§How it works

The lifecycle has three parts:

Name each store. The application gives every persistent store a permanent identity, such as app.users.v1.
Remember its location. On the first successful installation, ic-memory records which storage location belongs to each name.
Check before opening. On every installation or upgrade, the new application version declares the layout it expects. ic-memory compares that layout with the remembered one before any application store opens.

The ic-memory lifecycle: name each store, remember its storage location, then compare the expected and remembered layouts before opening data. Matching layouts open safely; conflicts stop with an error.

Name stores, retain their locations, and check the complete layout before opening data.

If the layouts agree, the application receives permission to open its stores. If a known name moved, a location changed owner, or another allocation rule is broken, the check returns an error and ic-memory grants no permission to open application stores.

This check-and-record operation is called bootstrap. Its important guarantee is validation before open: a diagnostic report or an uncommitted validation result cannot grant access to a store. The runtime publishes permission only after the allocation ledger has been recovered, checked, and durably updated.

The ledger stores one current record per allocated identity and the latest schema metadata. It keeps no per-upgrade or schema-change history. Omitted and retired identities retain their IDs to prevent accidental reuse. A commit counter and two protected commit slots support stale-proof checks and corruption detection.

§What happens when a check fails?

Bootstrap returns an error before application stores are opened. ic-memory does not silently repair, move, discard, or reinterpret a conflicting allocation. A failed attempt publishes no permission to open stores.

Treat the error as an upgrade-safety signal: keep the existing stable memory, inspect the requested keys, host namespace grants, exclusions, policy and diagnostics, then correct the new application version. Do not erase the allocation ledger or replace it with an empty one to make the error disappear. See the symptom-based troubleshooting guide for safe next steps.

§What it protects

ic-memory protects store-to-location mappings, prevents slot reuse, validates upgrade layouts and component ownership, and checks layouts before opening. Applications still own backups, schema migrations, stored-data semantics, authorization, and disaster recovery.

Allocation safety complements backups, schema migration, authorization, and disaster recovery.

Retiring a store does not make its old location reusable. That tombstone is intentional: reusing the location could make a rollback open unrelated data.

§Terminology

A stable key is a permanent store name, such as app.orders.v1.
A memory ID is the physical MemoryManager location behind that key.
An authority names the linked component requesting a key.
A namespace grant admits that authority’s keys under a host-selected prefix.
The allocation pool contains eligible application IDs shared by all owners.
Bootstrap recovers, admits, resolves, validates and commits before opening.
The allocation ledger retains key-to-ID bindings and retirement tombstones.

The application owns one pool and one bootstrap per backing memory. Libraries contribute key requests, verify that the host included them, and open their committed keys. Owner labels are current host policy; they are not persisted ownership history, caller authentication or a sandbox for linked code.

§Quick start for developers

Add the crate:

[dependencies]
ic-memory = "0.34.0"

ic-memory re-exports its exact ic-stable-structures dependency through ic_memory::ic_stable_structures. Import collections, backing memories, and traits through that namespace so their types match the runtime:

use ic_memory::{
    RuntimeMemory,
    ic_stable_structures::{Cell, DefaultMemoryImpl},
};

type CounterStore = Cell<u64, RuntimeMemory<DefaultMemoryImpl>>;

A separate ic-stable-structures dependency is unnecessary for those imports.

§Declare keys; let the host allocate

Components name their owner and permanent keys without choosing IDs:

const MEMORY_AUTHORITY: &str = "example_app";
ic_memory::ic_memory_declaration!(
    authority = MEMORY_AUTHORITY,
    key = "example_app.users.v1",
);

fn initialize_stable_storage() -> Result<(), Box<dyn std::error::Error>> {
    let pool = ic_memory::MemoryAllocationPool::new(
        vec![ic_memory::MemoryAuthority::new(MEMORY_AUTHORITY, "example_app.")?],
        vec![],
    )?;
    ic_memory::bootstrap_default_memory_manager(&pool)?;
    let users = ic_memory::open_default_memory_manager_memory("example_app.users.v1")?;
    drop(users);
    Ok(())
}

Call this host bootstrap from initialization and post-upgrade before deferred stable collections open. Hosts needing admission or custom bucket geometry use the policy or configuration bootstrap helpers with the same explicit pool.

Known keys retain their committed IDs. New requests are sorted by key and take the lowest unclaimed ID in the common pool. Current, omitted, reserved and retired records occupy their IDs permanently. Namespace grants do not divide space between components. Exhaustion rejects the entire attempt.

See examples/key_only.rs and examples/composed_host.rs.

§Applications composed from several libraries

Every linked crate contributes requests to one immutable registry. The host admits disjoint namespaces for their named owners and supplies explicit physical exclusions for unmanaged MemoryManager clients. All components draw from the same remaining pool. The application bootstraps the combined layout once.

Libraries adopting an already bootstrapped host can verify that all of their requirements were included without rerunning bootstrap or replacing the host’s policy:

fn adopt_host() -> Result<(), Box<dyn std::error::Error>> {
    let requirements = ic_memory::sealed_declaration_snapshot()?;
    ic_memory::verify_default_memory_manager_authority(
        &requirements,
        "library_name",
    )?;
    Ok(())
}

Duplicate keys, overlapping namespace grants, foreign owner claims and excluded historical IDs fail before application stores open. Populated IDs without a ledger record must be explicitly excluded by the host; bootstrap refuses to infer custody from stored bytes.

The default runtime reserves IDs 0..=9 and keys under ic_memory.* for its own governance records. ID 0 contains the allocation ledger. Application code may use IDs through 254; ID 255 is the upstream unallocated sentinel and can never be declared.

§Stable keys

A stable key names the durable identity of a store, not its current numeric location:

namespace.component.store_or_role.vN

use ic_memory::StableKey;

StableKey::parse("app.orders.v1").expect("app key");
StableKey::parse("myapp.audit_log.v1").expect("app key");
StableKey::parse("icydb.test_db.users.data.v1").expect("database key");

Changing a key creates a new allocation identity. If the durable store is still the same store, keep its key and change its optional diagnostic schema metadata instead.

§Frequently asked questions

Does ic-memory move or migrate application data?

No. It validates allocation identity. Schema and data migrations remain the application’s responsibility.

Does it back up stable memory or recover corrupted application data?

No. Keep a separate backup and disaster-recovery plan. ic-memory fails closed when its allocation metadata cannot be recovered safely.

What happens when validation fails?

Bootstrap returns an error and publishes no open capability. Fix the proposed layout or policy; do not erase the ledger to bypass the conflict.

Can a retired memory location be reused?

No. Retirement is a permanent tombstone so a future version or rollback cannot mistake unrelated data for the retired store.

Does every library bootstrap separately?

No. One owner bootstraps each concrete runtime. Libraries verify their requirements against the host’s committed layout and open only their keys.

What does each library declare?

Each library declares permanent keys and its owner label. The host admits the namespace and owns physical exclusions; Memory assigns IDs and retains them.

Can an upgrade add a new store safely?

Yes, provided its key is new, the host admits its namespace and owner, the common pool has space, and the complete layout passes admission and validation.

§Operations and advanced integration

Operations and diagnostics explains memory attribution, bucket configuration, and runtime reports.
Troubleshooting maps common symptoms to safe recovery steps.
Advanced ic-memory covers explicit runtimes, custom policies, recovery, and manual bootstrap.
Safety invariants records the guarantees future changes must preserve.
Documentation index separates current guidance from historical engineering evidence.
Release guide is for maintainers preparing and publishing a release.

ic-memory is pre-1.0 infrastructure extracted from Canic. Current releases use the current API and durable format only; earlier pre-1.0 wire formats are not a compatibility target. Stable-memory allocation-governance primitives for Internet Computer canister upgrades.

ic-memory prevents stable-memory slot drift.

Once a stable key is committed to a physical allocation slot, future binaries must either reopen that same stable key on that same slot or declare a new stable key.

The crate records and validates durable ownership in both directions: an active stable key cannot move to a different physical slot, and an active physical slot cannot be reused by a different stable key.

The intended runtime integration flow is:

Recover the persisted allocation ledger.
Admit consumer identity and authorized historical selections from bounded metadata under the host’s RuntimeBootstrapPolicy.
Resolve key requests in the host-wide pool under current namespace grants, then validate retained key-to-ID bindings and application policy.
Stage and durably persist the next generation.
Only then open stable-memory handles through committed allocation authority.

This crate owns allocation invariants, not framework policy. Namespace rules, controller authorization, endpoint lifecycle, schema migrations, and application validation belong to the framework or application.

The host supplies one MemoryAllocationPool with explicit namespace grants and physical exclusions. Linked components request permanent keys and name their owner; they do not select numeric IDs or reserve component subranges. Namespace grants are current host policy, not caller authentication or persisted ownership labels. Application admission remains host-owned.

Use these primitives before opening stable-memory handles. Integrations should recover the historical ledger, declare the stores expected by the current binary, admit recovered identity, resolve requests, validate against history and policy, persist a new generation, and only then publish authority before opening slots through the storage owner.

Bounded physical attribution is available through MemoryRuntime::memory_allocations and default_memory_manager_memory_allocations. It reports actual persisted buckets and explicit residuals without decoding retained ownership. Virtual extent is not payload occupancy. Opens return RuntimeMemory; explicit MemoryManagerConfig selects fresh-state buckets or checks a persisted setting without migration. The default remains 128 pages.

MemoryRuntime is the canonical owner for one backing memory instance. It owns that memory’s manager, ledger persistence, bootstrap lifecycle, committed capability, opens, and diagnostics. Each bootstrap attempt fallibly decodes its ledger record and uses a temporary cell for capacity-checked writes. Linked code contributes declarations to one immutable SealedDeclarationSnapshot, which is supplied to each runtime independently.

AllocationBootstrap is the golden path for whichever layer owns a given ledger store. Canic may own bootstrap for a framework canister and compose IcyDB/application declarations through its registry; IcyDB may own bootstrap directly for generated database stores; or a standalone application canister may own bootstrap itself. Exactly one owner should bootstrap one ledger store. Multiple layers in the same canister must either compose declarations into that owner or use distinct ledger stores and allocation domains.

ic-stable-structures MemoryManager IDs are the first-class supported physical slot substrate. That ID domain is u8: IDs 0..=254 are usable, and ID 255 is always the ic-stable-structures unallocated sentinel. The crate still keeps narrow internal abstractions for storage adapters and diagnostics, but the native IC path is MemoryManager ID 0 -> ic-stable-structures::Cell<StableCellLedgerRecord, _> -> LedgerCommitStore -> CommittedGenerationBytes -> LedgerPayloadEnvelope -> RecoveredLedger -> ValidatedAllocations -> CommittedAllocations.

ic_stable_structures re-exports the exact substrate version used by this crate. Use its collections and traits with RuntimeMemory handles; ic-memory owns allocation governance without wrapping typed collections.

Re-exports§

pub use ic_stable_structures;

Macros§

eager_init: Register one pre-bootstrap hook.
ic_memory_declaration: Register a MemoryManager allocation declaration during static initialization.
ic_memory_key: Declare a key-only request and open it after the host has committed bootstrap.

Structs§

AllocationBootstrap: AllocationBootstrap
AllocationDeclaration: AllocationDeclaration
AllocationLedger: AllocationLedger
AllocationRecord: AllocationRecord
AllocationRetirement: AllocationRetirement
BootstrapAdmission: BootstrapAdmission
CommitStoreDiagnostic: CommitStoreDiagnostic
CommittedAllocations: CommittedAllocations
CommittedGenerationBytes: CommittedGenerationBytes
DeclarationSnapshot: DeclarationSnapshot
DiagnosticExport: DiagnosticExport
DiagnosticFailure: DiagnosticFailure
DiagnosticMemorySize: DiagnosticMemorySize
DiagnosticRecord: DiagnosticRecord
DiagnosticRuntimeBinding: DiagnosticRuntimeBinding
DiagnosticStableCell: DiagnosticStableCell
DualCommitStore: DualCommitStore
GenericAllocationPolicy: GenericAllocationPolicy
LedgerCommitStore: LedgerCommitStore
LedgerPayloadEnvelope: LedgerPayloadEnvelope
MemoryAllocation: MemoryAllocation
MemoryAllocationPool: MemoryAllocationPool
MemoryAllocationSummary: MemoryAllocationSummary
MemoryAllocations: MemoryAllocations
MemoryAuthority: MemoryAuthority
MemoryBindingSummary: MemoryBindingSummary
MemoryManagerConfig: MemoryManagerConfig
MemoryManagerIdRange: MemoryManagerIdRange
MemoryManagerSlot: MemoryManagerSlot
MemoryRequest: MemoryRequest
MemoryRuntime: MemoryRuntime
MemoryRuntimeDoctorReport: MemoryRuntimeDoctorReport
PendingBootstrapCommit: PendingBootstrapCommit
PolicyIdentity: PolicyIdentity
RecoveredAllocationMetadata: RecoveredAllocationMetadata
RecoveredLedger: RecoveredLedger
RuntimeMemory: RuntimeMemory
SchemaMetadata: SchemaMetadata
SealedDeclarationFingerprint: SealedDeclarationFingerprint
SealedDeclarationSnapshot: SealedDeclarationSnapshot
StableCellLedgerRecord: StableCellLedgerRecord
StableKey: StableKey
StableKeyError: StableKeyError
ValidatedAllocations: ValidatedAllocations

Enums§

AllocationBinding: AllocationBinding
AllocationReservationError: AllocationReservationError
AllocationRetirementError: AllocationRetirementError
AllocationStageError: AllocationStageError
AllocationState: AllocationState
AllocationValidationError: AllocationValidationError
BootstrapAdmissionError: BootstrapAdmissionError
BootstrapError: BootstrapError
BootstrapReservationError: BootstrapReservationError
BootstrapRetirementError: BootstrapRetirementError
CommitRecoveryError: CommitRecoveryError
CommitSlotDiagnostic: CommitSlotDiagnostic
DeclarationSnapshotError: DeclarationSnapshotError
DiagnosticCheck: DiagnosticCheck
DiagnosticCode: DiagnosticCode
DiagnosticStableCellStatus: DiagnosticStableCellStatus
LedgerCommitError: LedgerCommitError
LedgerIntegrityError: LedgerIntegrityError
LedgerPayloadEnvelopeError: LedgerPayloadEnvelopeError
MemoryAllocationPoolError: MemoryAllocationPoolError
MemoryManagerLayoutError: MemoryManagerLayoutError
MemoryManagerRangeError: MemoryManagerRangeError
MemoryManagerSlotError: MemoryManagerSlotError
MemoryResolutionError: MemoryResolutionError
PolicyIdentityError: PolicyIdentityError
RuntimeAdoptionError: RuntimeAdoptionError
RuntimeBootstrapError: RuntimeBootstrapError
RuntimeConstructionError: RuntimeConstructionError
RuntimeDiagnosticError: RuntimeDiagnosticError
RuntimeGrowError: RuntimeGrowError
RuntimeOpenError: RuntimeOpenError
RuntimeStateError: RuntimeStateError
SchemaMetadataError: SchemaMetadataError
StableCellLedgerError: StableCellLedgerError
StableCellPayloadError: StableCellPayloadError
StaticMemoryDeclarationError: StaticMemoryDeclarationError

Constants§

IC_MEMORY_AUTHORITY_OWNER: Diagnostic owner label for ic-memory allocation-governance infrastructure.
IC_MEMORY_LEDGER_LABEL: Diagnostic label of the allocation ledger when backed by the current MemoryManager substrate.
IC_MEMORY_LEDGER_STABLE_KEY: Stable key of the allocation ledger when backed by the current MemoryManager substrate.
IC_MEMORY_STABLE_KEY_PREFIX: Stable-key namespace prefix reserved for ic-memory allocation-governance infrastructure.
LEDGER_PAYLOAD_FORMAT_VERSION: Current durable ledger payload format version.
MAX_LEDGER_BYTES: Maximum encoded ownership ledger size (64 KiB for at most 255 records).
MAX_LEDGER_NESTING: Maximum CBOR container nesting on maintained decode paths.
MAX_LEDGER_RECORD_BYTES: Two bounded CBOR byte strings plus record metadata (128 KiB + 4 KiB).
MEMORY_MANAGER_GOVERNANCE_MAX_ID: Last MemoryManager ID reserved for ic-memory governance in the current substrate.
MEMORY_MANAGER_INVALID_ID: MemoryManager unallocated-bucket sentinel. This is not a usable slot.
MEMORY_MANAGER_LEDGER_ID: MemoryManager ID used by the allocation ledger in the current MemoryManager substrate.
MEMORY_MANAGER_MAX_ID: Last usable MemoryManager virtual memory ID.
MEMORY_MANAGER_MIN_ID: First usable MemoryManager virtual memory ID.
STABLE_CELL_HEADER_SIZE: Stable-cell header byte length.
STABLE_CELL_LAYOUT_VERSION: Stable-cell layout version supported by this adapter.
STABLE_CELL_MAGIC: Stable-cell magic prefix written by ic-stable-structures::Cell.
STABLE_CELL_VALUE_OFFSET: Byte offset where the stable-cell value payload starts.
WASM_PAGE_SIZE_BYTES: WebAssembly page size used by ic-stable-structures memory implementations.

Traits§

AllocationPolicy: AllocationPolicy
RuntimeBootstrapPolicy: RuntimeBootstrapPolicy

Functions§

bootstrap_default_memory_manager: Bootstrap this thread’s default runtime using the host pool and built-in allocation policy.
bootstrap_default_memory_manager_with_config: Bootstrap the default runtime with an explicit bucket setting and allocation policy.
bootstrap_default_memory_manager_with_policy: Bootstrap this thread’s default runtime with caller-supplied policy.
committed_allocations: Return this thread’s default runtime committed allocation capability.
decode_stable_cell_ledger_record: Decode a StableCellLedgerRecord from stable-cell value bytes.
decode_stable_cell_ledger_record_from_memory: Fallibly decode a ledger record from its stable-cell envelope and value.
decode_stable_cell_payload: Decode the raw value payload from an ic-stable-structures::Cell memory.
default_memory_manager_commit_recovery_diagnostic: Diagnose protected commit recovery for this thread’s default runtime.
default_memory_manager_diagnostic_export: Export this thread’s default runtime ledger and live memory sizes.
default_memory_manager_doctor_report: Build preflight and lifecycle diagnostics for this thread’s default runtime.
default_memory_manager_doctor_report_with_policy: Build diagnostics for this thread’s default runtime under one explicit policy.
default_memory_manager_memory_allocation_summary: Measure numeric allocation totals in the existing default runtime.
default_memory_manager_memory_allocations: Measure the existing default runtime without constructing a manager or initializing backing memory.
default_memory_manager_memory_id: Resolve an application key’s committed ID in the existing default runtime. Does not construct a manager, open memory, or choose a bucket configuration.
is_default_memory_manager_bootstrapped: Return whether this thread’s default runtime has completed bootstrap.
is_ic_memory_stable_key: Return true when stable_key belongs to the ic-memory namespace.
memory_manager_governance_range: MemoryManager range reserved for ic-memory governance in the current substrate.
open_default_memory_manager_memory: Open a committed memory from this thread’s default runtime. Does not construct an absent runtime or select its bucket configuration.
register_memory_request: Register a key-only request before the linked snapshot seals.
sealed_declaration_snapshot: Seal and return the canonical linked-program declaration snapshot.
validate_allocations: Validate a committed ledger and current declarations before opening.
validate_memory_manager_id: Validate that a MemoryManager ID is usable as an allocation slot.
verify_default_memory_manager_authority: Verify one consumer’s allocation requirements against the existing host runtime. Does not construct, bootstrap, replay admission or change configuration.

Crate ic_memory

Crate ic_memory Copy item path

§What it does

§Why this matters

§Is it useful for my application?

§How it works

§What happens when a check fails?

§What it protects

§Terminology

§Quick start for developers

§Declare keys; let the host allocate

§Applications composed from several libraries

§Stable keys

§Frequently asked questions

§Operations and advanced integration

Re-exports§

Macros§

Structs§

Enums§

Constants§

Traits§

Functions§

Crate ic_memory