Crate hashtree_lmdb

Expand description

§hashtree-lmdb

LMDB-backed content-addressed blob storage for hashtree.

High-performance storage backend using LMDB (Lightning Memory-Mapped Database) for fast key-value storage.

§Write and reopen a file

cargo add hashtree-core hashtree-lmdb tempfile
cargo add tokio --features macros,rt-multi-thread

This runnable example uses a temporary directory, deleted on exit. In an app, use your persistent data directory and keep the root metadata there too.

use hashtree_core::{Cid, HashTree, HashTreeConfig};
use hashtree_lmdb::LmdbBlobStore;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let directory = tempfile::tempdir()?;
    let blobs = directory.path().join("blobs");
    let root_file = directory.path().join("root.txt");
    {
        let store = Arc::new(LmdbBlobStore::new(&blobs)?);
        let tree = HashTree::new(HashTreeConfig::new(store));
        let (cid, _) = tree.put(b"Persistent data").await?;
        std::fs::write(&root_file, cid.to_string())?;
    } // Drop the tree and store before reopening the database.

    let store = Arc::new(LmdbBlobStore::new(&blobs)?);
    let tree = HashTree::new(HashTreeConfig::new(store));
    let cid = Cid::parse(&std::fs::read_to_string(root_file)?)?;
    let bytes = tree.get(&cid, Some(1024)).await?.ok_or("File is unavailable")?;
    assert_eq!(bytes, b"Persistent data");
    Ok(())
}

The root file includes the decryption key: restrict access to it. In a real app, write new root metadata atomically after the blocks are durable, retain old roots as needed, and coordinate pin/eviction policy. Saving only a hash loses encrypted content’s key. The example owns its database; shared application stores should use the canonical opener below.

Trusted processes running as the same OS user may open one store path on a local filesystem directly; no storage daemon is required. LMDB serializes writers and exposes only committed transactions to readers. Blob insertion uses NO_OVERWRITE, so concurrent writes of one correctly computed hash are idempotent. An uncommitted transaction is rolled back if its process dies. This guarantee does not extend to network filesystems or to a separate application’s mutable metadata databases.

Processes sharing Hashtree’s application store should not construct LmdbBlobStore from data_dir/blobs themselves. Use the canonical opener with the same configured storage budget as the application:

use hashtree_core::StoreError;
use hashtree_lmdb::{open_shared_lmdb_blob_store, ConfiguredLmdbBlobStore};
use std::{path::Path, sync::Arc};

fn open_app_store(data_dir: &Path, max_size_bytes: u64) -> Result<Arc<ConfiguredLmdbBlobStore>, StoreError> {
    Ok(Arc::new(open_shared_lmdb_blob_store(data_dir, max_size_bytes)?))
}

The returned ConfiguredLmdbBlobStore implements Store. A fresh shared store is a PoolStore with a persistent catalog and one default member. Existing single LMDB stores remain single stores instead of being silently reclassified. The former hot/legacy environment-variable bridge has been removed. Migrate each old LMDB source into the pool with htree storage pool migrate-lmdb before cutting applications over to the pool.

The pool is one application-owned write destination. Its LMDB members are opaque storage resources, and the pool exclusively chooses one member for every blob. It persists exact hash placement and member identity, centrally verifies SHA-256 on writes, reads, and moves, and keeps pin/access metadata in the shared catalog. Read-only BlobRouter code treats the complete pool as one route; it does not select pool members.

Member configuration is explicit and persisted. Operators can override:

logical capacity, which controls capacity-proportional placement;
LMDB map size, independently of capacity for external-pack-backed members;
external pack directory, spill threshold, pack target, and sync behavior; and
per-member read and write concurrency limits.

The automatically created first member starts with 10 GiB of bounded staging headroom above the application’s quota. That keeps placement capacity distinct from application retention while a tree is being written and indexed. It is an initial guardrail only; operators should configure each member’s actual logical capacity when adding or resizing storage.

These are guardrails, not media classes. The pool does not label a path SSD, HDD, local, or remote. Per-process latency, throughput, reliability, cooldown, and occasional exploration influence placement among eligible members, but that bounded learning is disposable and correctness never depends on it.

Use the CLI to inspect and change membership:

htree storage pool status
htree storage pool add PATH --capacity-gb N [member overrides]
htree storage pool configure MEMBER_ID [--capacity-gb N] [--max-reads N] [--max-writes N]
htree storage pool drain MEMBER_ID
htree storage pool maintain --max-items N
htree storage pool remove MEMBER_ID

Planned removal is drain, repeated bounded maintain, then remove. Every move is read, hash-verified, written, hash-verified, catalog-committed, and only then deleted from the source. remove refuses a non-empty member. Sudden loss is not presented as redundancy: hashes placed only on the unavailable member fail until that exact member identity returns or another source can supply the bytes.

The immutable blob invariant is narrower than the mutable application state:

LmdbBlobStore callers must provide the SHA-256 key for bytes they write. The pooled store recomputes it at its boundary, while retrieval layers still hash-verify any bytes received from another route before return or caching.
Access order, aggregate statistics, pin counts, and garbage collection changes are transactional shared state. Pin counts are durable application references, not process leases: a process dying does not undo a committed pin.
Processes allowed to run garbage collection must use one agreed quota policy because eviction affects every process sharing the catalog. Placement capacity is not a second application-level quota or write policy.
Every process must use the canonical opener. Stable identity markers prevent a missing mount or newly recreated directory from masquerading as a member.

All simultaneously running processes must open each member with the same, sufficient map size. If another process enlarges a map after a process has opened it, that older handle reports MDB_MAP_RESIZED; it must exit and reopen before accessing the enlarged environment. Establish a larger map before starting the other processes. Membership and concurrency changes are generation-tracked and refresh safely across live processes; LMDB map resizing is deliberately not a live membership operation.

On Darwin, LMDB’s default System V semaphore backend uses SEM_UNDO, and the default per-process kern.sysv.semume limit is ten. The bound applies to simultaneous lock-owning transactions across distinct environments, not to the number of configured pool members. Hashtree therefore coordinates its blob and mutable-metadata LMDB wrappers through one process-wide gate: up to four managed write transactions may overlap, while short read/open lock acquisition is serialized and headroom remains for the social-graph backend. Linux and Windows do not receive this Darwin-only bound. Storage membership remains scalable, and no daemon or operator sysctl change is required.

For online imports, htree storage pool migrate-lmdb opens the source read-only, never resizes it, scans in bounded hash order, and writes only through the target pool. Persist its cursor with --state-file; replay is idempotent. Because a live source can add a hash before the saved cursor, run another complete pass after stopping source writes, then verify the final destination before cutover.

§Features

Memory-mapped I/O for fast reads
ACID transactions
Multiprocess-safe committed reads and idempotent blob writes
Crash recovery for unfinished LMDB transactions

Part of hashtree-rs.

Structs§

ExternalBlobOptions: Options for storing larger LMDB blobs in normal files.
LmdbBlobReader: Read-only view of an existing LMDB blob store for online migration and verification.
LmdbBlobStore: LMDB-backed blob store implementing hashtree’s Store trait.
LmdbEnvironmentGeneration: Stable LMDB generation fields used to bind a source-terminal receipt.
LmdbParallelRawKeyScanner: Ordered, bounded-memory raw-key scanner backed by concurrent LMDB cursors.
LmdbSourceAuditBatch
LmdbSourceKeyAudit: Exhaustive proof of the ordered raw blob keys in a frozen migration source.
LmdbSourceKeyAuditBuilder: Streaming builder for LmdbSourceKeyAudit.
LmdbSourceKeysetAudit: Exhaustive key-set proof for a terminal LMDB migration source.
LmdbStats
PhysicalSpaceGuard
PinnedLmdbFileIdentity: Caller-pinned identity of one LMDB file.
PinnedLmdbIdentity: Caller-pinned identities for both files in an LMDB environment.
PoolDeleteProtectionChange
PoolDeleteProtectionGuard: An exact durable delete-protection record held against concurrent release. Dropping this value releases only the coordination lock, not the durable protection record.
PoolDeleteProtectionStatus: One durable, non-expiring guard that keeps PoolStore writable while rejecting logical deletion. The exact catalog record identity is required to release it, preventing a stale operator command from removing a newer guard.
PoolMaintenanceReport
PoolManifestIdentity: Identity of the complete Pool manifest observed by a read-only reader.
PoolMemberConfig: Persistent configuration for one opaque LMDB storage member.
PoolMemberId: Stable identity for one storage member in a local blob pool.
PoolMemberRuntimePaths
PoolMemberStatus
PoolMigrationAuditStore: Durable, provenance-separated source reconciliation and target body proofs.
PoolMigrationAuditSummary: Exact summary of a durable online migration verified set.
PoolMigrationBatch
PoolPhysicalAudit: Exact catalog-to-physical-member proof that does not reread blob payloads.
PoolReadBatchItem: One result from a bounded, strictly read-only Pool batch.
PoolStalePending: One exact catalog record authorized for stale-Pending crash recovery.
PoolStalePendingCleanupReport: Result of one exact stale-Pending recovery transaction.
PoolStore
PoolStoreConfig
PoolStoreReader: Strictly read-only view of a PoolStore for exhaustive validation.
PoolTemperatureConfig
PoolTemperatureReport
PoolTerminalAudit: Exhaustive terminal-state proof for a controlled Pool.
ReadOnlyPoolCatalogAudit
ReadOnlyPoolManifestMember: One exact member record from a read-only Pool manifest snapshot.
ReadOnlyPoolManifestSnapshot: The exact stored Pool manifest observed while opening a read-only reader.
ReadOnlyPoolStore: Strictly read-only view of an existing PoolStore catalog and its members.

Enums§

ConfiguredLmdbBlobStore: One canonical application-owned LMDB blob store.
PoolCatalogLocation: Exact, read-only Pool catalog state for one blob.
PoolMemberState

Constants§

DEFAULT_POOL_MIGRATION_MAX_BUFFER_BYTES: Default upper bound for complete blob payloads retained by a migration.
LOCAL_ADD_EXTERNAL_BLOB_DIR_NAME
MAX_GUARDED_WRITE_BYTES
MAX_PARALLEL_RAW_KEY_SCAN_CONCURRENCY: Hard cap for concurrent value-free LMDB key-range cursors.
PHYSICAL_SPACE_METADATA_MARGIN
POOL_AUDIT_READ_ONLY_ENV
POOL_AUDIT_READ_ONLY_ERROR
POOL_DELETE_PROTECTED
SHARED_BLOB_MIN_MAP_SIZE_BYTES
SHARED_BLOB_POOL_DIR_NAME

Functions§

audit_lmdb_source_batch_with_max_buffer_bytes: Read and content-hash one bounded source page without opening or mutating a target. This is the expensive source proof used before final downtime.
compute_sha256: Compute SHA256 hash of data
migrate_lmdb_batch: Copy and verify one bounded lexicographic page from an existing LMDB store.
migrate_lmdb_batch_with_max_buffer_bytes: Copy and verify one lexicographic cursor page using byte-bounded writes.
migrate_lmdb_batch_with_max_buffer_bytes_and_authorizer: Copy and verify one lexicographic cursor page, authorizing each bounded target-write group immediately before it can mutate the Pool.
migrate_lmdb_hashes_with_max_buffer_bytes_and_authorizer: Verify and migrate an explicit sorted source hash set.
open_configured_lmdb_blob_store: Open one existing unbounded LMDB environment for a canonical blob path.
open_shared_lmdb_blob_store: Open Hashtree’s canonical same-user LMDB blob layout for a data directory.
open_shared_lmdb_blob_store_with_space_guard: Open the same canonical layout with process-local physical write admission.
pool_audit_read_only_enabled
reconcile_lmdb_source_batch_with_max_buffer_bytes_and_authorizer: Reconcile one sorted source page using source keys and target metadata.
reconcile_lmdb_source_entries_with_max_buffer_bytes_and_authorizer: Reconcile an exact, receipt-proven sorted hash/size page.
reconcile_lmdb_source_hashes_with_max_buffer_bytes_and_authorizer: Reconcile exact ordered source keys, deriving sizes from the target when already Stored and reading source bodies only for Missing/Pending.
reconcile_lmdb_source_union_page_with_max_buffer_bytes_and_authorizer: Reconcile one globally sorted, receipt-proven union page.