hashtree-lmdb

LMDB-backed content-addressed blob storage for hashtree.

High-performance storage backend using LMDB (Lightning Memory-Mapped Database) for fast key-value storage.

Published API · Core guide

Write and reopen a file

cargo add hashtree-core hashtree-lmdb tempfile
cargo add tokio --features macros,rt-multi-thread

This runnable example uses a temporary directory, deleted on exit. In an app, use your persistent data directory and keep the root metadata there too.

use hashtree_core::{Cid, HashTree, HashTreeConfig};
use hashtree_lmdb::LmdbBlobStore;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let directory = tempfile::tempdir()?;
    let blobs = directory.path().join("blobs");
    let root_file = directory.path().join("root.txt");
    {
        let store = Arc::new(LmdbBlobStore::new(&blobs)?);
        let tree = HashTree::new(HashTreeConfig::new(store));
        let (cid, _) = tree.put(b"Persistent data").await?;
        std::fs::write(&root_file, cid.to_string())?;
    } // Drop the tree and store before reopening the database.

    let store = Arc::new(LmdbBlobStore::new(&blobs)?);
    let tree = HashTree::new(HashTreeConfig::new(store));
    let cid = Cid::parse(&std::fs::read_to_string(root_file)?)?;
    let bytes = tree.get(&cid, Some(1024)).await?.ok_or("File is unavailable")?;
    assert_eq!(bytes, b"Persistent data");
    Ok(())
}

The root file includes the decryption key: restrict access to it. In a real app, write new root metadata atomically after the blocks are durable, retain old roots as needed, and coordinate pin/eviction policy. Saving only a hash loses encrypted content's key. The example owns its database; shared application stores should use the canonical opener below.

Multiprocess sharing

Trusted processes running as the same OS user may open one store path on a local filesystem directly; no storage daemon is required. LMDB serializes writers and exposes only committed transactions to readers. Blob insertion uses NO_OVERWRITE, so concurrent writes of one correctly computed hash are idempotent. An uncommitted transaction is rolled back if its process dies. This guarantee does not extend to network filesystems or to a separate application's mutable metadata databases.

Processes sharing Hashtree's application store should not construct LmdbBlobStore from data_dir/blobs themselves. Use the canonical opener with the same configured storage budget as the application:

use hashtree_core::StoreError;
use hashtree_lmdb::{open_shared_lmdb_blob_store, ConfiguredLmdbBlobStore};
use std::{path::Path, sync::Arc};

fn open_app_store(data_dir: &Path, max_size_bytes: u64) -> Result<Arc<ConfiguredLmdbBlobStore>, StoreError> {
    Ok(Arc::new(open_shared_lmdb_blob_store(data_dir, max_size_bytes)?))
}

The returned ConfiguredLmdbBlobStore implements Store. A fresh shared store is a PoolStore with a persistent catalog and one default member. Existing single LMDB stores remain single stores instead of being silently reclassified. The former hot/legacy environment-variable bridge has been removed. Migrate each old LMDB source into the pool with htree storage pool migrate-lmdb before cutting applications over to the pool.

The pool is one application-owned write destination. Its LMDB members are opaque storage resources, and the pool exclusively chooses one member for every blob. It persists exact hash placement and member identity, centrally verifies SHA-256 on writes, reads, and moves, and keeps pin/access metadata in the shared catalog. Read-only BlobRouter code treats the complete pool as one route; it does not select pool members.

Member configuration is explicit and persisted. Operators can override:

• logical capacity, which controls capacity-proportional placement;
• LMDB map size, independently of capacity for external-pack-backed members;
• external pack directory, spill threshold, pack target, and sync behavior; and
• per-member read and write concurrency limits.

The automatically created first member starts with 10 GiB of bounded staging headroom above the application's quota. That keeps placement capacity distinct from application retention while a tree is being written and indexed. It is an initial guardrail only; operators should configure each member's actual logical capacity when adding or resizing storage.

These are guardrails, not media classes. The pool does not label a path SSD, HDD, local, or remote. Per-process latency, throughput, reliability, cooldown, and occasional exploration influence placement among eligible members, but that bounded learning is disposable and correctness never depends on it.

Use the CLI to inspect and change membership:

htree storage pool status
htree storage pool add PATH --capacity-gb N [member overrides]
htree storage pool configure MEMBER_ID [--capacity-gb N] [--max-reads N] [--max-writes N]
htree storage pool drain MEMBER_ID
htree storage pool maintain --max-items N
htree storage pool remove MEMBER_ID

Planned removal is drain, repeated bounded maintain, then remove. Every move is read, hash-verified, written, hash-verified, catalog-committed, and only then deleted from the source. remove refuses a non-empty member. Sudden loss is not presented as redundancy: hashes placed only on the unavailable member fail until that exact member identity returns or another source can supply the bytes.

The immutable blob invariant is narrower than the mutable application state:

• LmdbBlobStore callers must provide the SHA-256 key for bytes they write. The pooled store recomputes it at its boundary, while retrieval layers still hash-verify any bytes received from another route before return or caching.
• Access order, aggregate statistics, pin counts, and garbage collection changes are transactional shared state. Pin counts are durable application references, not process leases: a process dying does not undo a committed pin.
• Processes allowed to run garbage collection must use one agreed quota policy because eviction affects every process sharing the catalog. Placement capacity is not a second application-level quota or write policy.
• Every process must use the canonical opener. Stable identity markers prevent a missing mount or newly recreated directory from masquerading as a member.

All simultaneously running processes must open each member with the same, sufficient map size. If another process enlarges a map after a process has opened it, that older handle reports MDB_MAP_RESIZED; it must exit and reopen before accessing the enlarged environment. Establish a larger map before starting the other processes. Membership and concurrency changes are generation-tracked and refresh safely across live processes; LMDB map resizing is deliberately not a live membership operation.

On macOS, LMDB's default System V semaphore backend uses SEM_UNDO, and the default per-process kern.sysv.semume limit is ten. The bound applies to simultaneous lock-owning transactions across distinct environments, not to the number of configured pool members. Hashtree therefore coordinates its blob and mutable-metadata LMDB wrappers through one process-wide gate: up to four managed write transactions may overlap, while short read/open lock acquisition is serialized and headroom remains for the social-graph backend. Linux and Windows do not receive this Darwin-only bound. Storage membership remains scalable, and no daemon or operator sysctl change is required.

On iOS, the crate selects LMDB's POSIX named-semaphore backend automatically: System V semaphore syscalls are unsupported there. Reader and writer locking, transactions, and explicit durability through force_sync() remain enabled. POSIX semaphores do not provide SEM_UNDO: if a process dies while holding a lock, every other process using that environment must exit before reopening it. LMDB then exclusively reinitializes the abandoned locks and recovers committed data. An iOS app and its extensions must coordinate this lifecycle, or use separate store paths; do not assume live-peer owner-death recovery. All processes sharing an environment must use the same locking backend.

For online imports, htree storage pool migrate-lmdb opens the source read-only, never resizes it, scans in bounded hash order, and writes only through the target pool. Persist its cursor with --state-file; replay is idempotent. Because a live source can add a hash before the saved cursor, run another complete pass after stopping source writes, then verify the final destination before cutover.

Features

• Memory-mapped I/O for fast reads
• ACID transactions
• Multiprocess-safe committed reads and idempotent blob writes
• Crash recovery for unfinished LMDB transactions

Part of hashtree-rs.