alkblobs

Content-addressed blob storage: a multi-hash put/get/verify store with pluggable backends, transport- and protocol-agnostic.

What it is (and is not)

alkblobs is the store layer of the alk* family: content-addressed bytes keyed under one canonical content hash — the git-family oid derivation, "blob <len>\0" + content, SHA-256 canonically with SHA-1 tolerated for existing repos — and a second derivation family for git's non-blob objects: the git-object family (registry 0x03/0x04, ADR-018) derives over git's loose-object form, so trees, commits, and tags store under their own oids through the derivation-explicit put seam (put_as(algorithm, len: Option<u64>, stream), ADR-020), byte-identical to git hash-object. Verification is built into the data path: the write verifies over the stored artifact as a whole before it publishes, and the kv tier's materialized reads verify against the key (ADR-006's verification-on-get posture); the large tier's windows serve streamed ranges with read_range returning a slice digest the caller carries out-of-band. The addressing vocabulary (Key, Digest) is family-explicit: addresses form only under a named derivation (Algorithm::derive), and pre-computed oids enter as typed keys over the tagged physical encoding (Key::from_bytes; ADR-020) — the seam alkgit-style consumers plan against.

It is not a network crate: no transport, no wire protocol in-core. The store never learns where bytes came from — an ops layer, a local writer, and a network push are all just calls to Store::put_as. The one network surface, the ops module (verified fetch/put/control ops over alkcall), rides behind the default-off ops feature, and its authorization rides alkcall's AccessControl seam — never an in-band scheme. The spec and ADR record is docs/architecture/; where this README and the specs disagree, the specs win.

Quick start

use alkblobs::{Algorithm, Store};

let store = Store::mem_only();

let content: &[u8] = b"hello, blobs";
let (key, pin) = store
    .put_as(
        Algorithm::GitBlobSha256,
        Some(content.len() as u64),
        std::io::Cursor::new(content),
    )
    .await
    .unwrap();

let (len, blob) = store.get(&key.as_bytes()).await.unwrap().unwrap();
assert_eq!(blob.read_at(0, len).await.unwrap(), content);

drop(pin);
let report = store.sweep().await.unwrap();
assert!(report.aborted.is_some()); // zero registered liveness sources: refuse-to-run, the safe default
assert_eq!(report.deleted, 0);

The Pin guard is the write path's protection — pin-before-publish, release on drop. The sweep above is GC's refuse-to-run posture: a sweep is a mark/delete-window run whose aborts are report data (SweepReport carries the counts either way), and with no liveness source registered it marks nothing. To actually let GC reap, register a liveness source — a consumer trait over your own reference state (LivenessRegistration::InProcess, or a fleet table reference for the fleet forms) — and let the Pin guards drop when your references do. Direct delete and the batch write seam (Batch, the multi-put pin group) round out the lifecycle.

The examples' error handling is .unwrap() for brevity; the facade is fallible end to end (Result, the crate's typed Error).

The default durable shape — dual-tier, kv = sqlite + large = local, size-threshold dispatch at the shipped 128 KiB default:

use alkblobs::{DEFAULT_DISPATCH_THRESHOLD, Store};
use std::time::Duration;

let store = Store::dual_tier_sqlite_local(
    "./blobs.db",
    "./blobs-root",
    DEFAULT_DISPATCH_THRESHOLD,
    Duration::from_secs(300), // the delete window's cadence knob
)
.await
.unwrap();

And one feature-gated variant — kv_only_postgres, with the postgres feature enabled (the fleet host's kv-only shape; over- threshold puts reject at put time, per the kv-only contract):

use alkblobs::{PgOpts, DEFAULT_DISPATCH_THRESHOLD, Store};
use std::time::Duration;

let store = Store::kv_only_postgres(
    "host=localhost user=postgres dbname=blobs",
    PgOpts::default(),
    DEFAULT_DISPATCH_THRESHOLD,
    Duration::from_secs(300),
)
.await
.unwrap();

The fleet shapes — DB-backed pins, the sweep lease, fleet: true declared explicitly — are further named constructors under the postgres feature (fleet state's host), the pg-lo pairings under pg-lo; the rustdoc's constructor table is the full surface.

Features

Features gate engines; tiers are constructor composition, not features (ADR-013 §4). Default builds are mem + sqlite:

Feature	Default	Pulls
`sqlite`	on	`rusqlite` (bundled)
`mem`	on	nothing
`postgres`	off	`tokio-postgres` + `deadpool-postgres`
`pg-lo`	off	the same pg driver cluster
`ops`	off	`alkcall`, `serde`, `serde_json`, `getrandom`

postgres and pg-lo are independent (a pg-pure build with sqlite off is supported, non-default); ops composes with anything.

Engines

One contract per tier, five shipped engines — docs/architecture/backends-and-dispatch.md is the table's home:

Tier	Engine	Feature	Durability posture
kv	sqlite	`sqlite`	WAL + `synchronous=NORMAL` on a single host; `busy_timeout=5000` contention posture pinned; single-writer ceiling
kv	postgres	`postgres`	pooled sessions over one server; per-session `synchronous_commit` knob; the fleet state's host
kv	mem	`mem`	in-process ephemeral — test/reference contract anchor, never production
large	local	(ungated)	flat sharded files; fsync-durable no-replace publish (ADR-015)
large	pg-lo	`pg-lo`	postgres Large Objects; windowed reads; one large-object consumer per database (ADR-016)

Engine selection has no benchmark claims beyond the recorded POC numbers' posture in docs/architecture/backends-and-dispatch.md and docs/research/: pick by deployment shape (single host / shared server / ephemeral, small regime / large regime / consolidation on one pg instance).

Documentation