alkblobs 0.2.2

Content-addressed blob storage: multi-hash put/get/verify store with pluggable backends, transport- and protocol-agnostic
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166

# alkblobs

Content-addressed blob storage: a multi-hash put/get/verify store with
pluggable backends, transport- and protocol-agnostic.

## What it is (and is not)

`alkblobs` is the store layer of the alk* family: content-addressed
bytes keyed under one canonical content hash — the git-family oid
derivation, `"blob <len>\0" + content`, SHA-256 canonically with SHA-1
tolerated for existing repos — and a second derivation family for
git's non-blob objects: the git-object family (registry `0x03`/`0x04`,
ADR-018) derives over git's loose-object form, so trees, commits, and
tags store under their own oids through the derivation-explicit put
seam (`put_as(algorithm, len: Option<u64>, stream)`,
[ADR-020](docs/architecture/decisions/020-family-explicit-wire-payloads.md)),
byte-identical to `git hash-object`. Verification is built into the
data path:
the write verifies over the stored artifact as a whole before it
publishes, and the kv tier's materialized reads verify against the key
(ADR-006's verification-on-get posture); the large tier's windows serve
streamed ranges with `read_range` returning a slice digest the caller
carries out-of-band. The addressing vocabulary (`Key`, `Digest`) is
family-explicit: addresses form only under a named derivation
(`Algorithm::derive`), and pre-computed oids enter as typed keys over
the tagged physical encoding (`Key::from_bytes`;
[ADR-020](docs/architecture/decisions/020-family-explicit-wire-payloads.md))
— the seam alkgit-style consumers plan against.

It is **not** a network crate: no transport, no wire protocol in-core.
The store never learns where bytes came from — an ops layer, a local
writer, and a network push are all just calls to `Store::put_as`.
The one
network surface, the `ops` module (verified fetch/put/control ops over
alkcall), rides behind the default-off `ops` feature, and its
authorization rides alkcall's `AccessControl` seam — never an in-band
scheme. The spec and ADR record is [docs/architecture/](docs/architecture/);
where this README and the specs disagree, the specs win.

## Quick start

```rust
use alkblobs::{Algorithm, Store};

let store = Store::mem_only();

let content: &[u8] = b"hello, blobs";
let (key, pin) = store
    .put_as(
        Algorithm::GitBlobSha256,
        Some(content.len() as u64),
        std::io::Cursor::new(content),
    )
    .await
    .unwrap();

let (len, blob) = store.get(&key.as_bytes()).await.unwrap().unwrap();
assert_eq!(blob.read_at(0, len).await.unwrap(), content);

drop(pin);
let report = store.sweep().await.unwrap();
assert!(report.aborted.is_some()); // zero registered liveness sources: refuse-to-run, the safe default
assert_eq!(report.deleted, 0);
```

The `Pin` guard is the write path's protection — pin-before-publish,
release on drop. The sweep above is GC's refuse-to-run posture: a sweep
is a mark/delete-window run whose aborts are report data (`SweepReport`
carries the counts either way), and with no liveness source registered
it marks nothing. To actually let GC reap, register a liveness source —
a consumer trait over your own reference state
(`LivenessRegistration::InProcess`, or a fleet table reference for the
fleet forms) — and let the `Pin` guards drop when your references do.
Direct `delete` and the batch write seam (`Batch`, the multi-put pin
group) round out the lifecycle.

The examples' error handling is `.unwrap()` for brevity; the facade is
fallible end to end (`Result`, the crate's typed `Error`).

The default durable shape — dual-tier, `kv = sqlite` + `large = local`,
size-threshold dispatch at the shipped 128 KiB default:

```rust
use alkblobs::{DEFAULT_DISPATCH_THRESHOLD, Store};
use std::time::Duration;

let store = Store::dual_tier_sqlite_local(
    "./blobs.db",
    "./blobs-root",
    DEFAULT_DISPATCH_THRESHOLD,
    Duration::from_secs(300), // the delete window's cadence knob
)
.await
.unwrap();
```

And one feature-gated variant — `kv_only_postgres`, with the
`postgres` feature enabled (the fleet host's kv-only shape; over-
threshold puts reject at put time, per the kv-only contract):

```rust
use alkblobs::{PgOpts, DEFAULT_DISPATCH_THRESHOLD, Store};
use std::time::Duration;

let store = Store::kv_only_postgres(
    "host=localhost user=postgres dbname=blobs",
    PgOpts::default(),
    DEFAULT_DISPATCH_THRESHOLD,
    Duration::from_secs(300),
)
.await
.unwrap();
```

The fleet shapes — DB-backed pins, the sweep lease, `fleet: true`
declared explicitly — are further named constructors under the
`postgres` feature (fleet state's host), the pg-lo pairings under
`pg-lo`; the rustdoc's constructor table is the full surface.

## Features

Features gate engines; tiers are constructor composition, not features
(ADR-013 §4). Default builds are `mem` + `sqlite`:

| Feature | Default | Pulls |
|---|---|---|
| `sqlite` | on | `rusqlite` (bundled) |
| `mem` | on | nothing |
| `postgres` | off | `tokio-postgres` + `deadpool-postgres` |
| `pg-lo` | off | the same pg driver cluster |
| `ops` | off | `alkcall`, `serde`, `serde_json`, `getrandom` |

`postgres` and `pg-lo` are independent (a pg-pure build with `sqlite`
off is supported, non-default); `ops` composes with anything.

## Engines

One contract per tier, five shipped engines —
[docs/architecture/backends-and-dispatch.md](docs/architecture/backends-and-dispatch.md)
is the table's home:

| Tier | Engine | Feature | Durability posture |
|---|---|---|---|
| kv | sqlite | `sqlite` | WAL + `synchronous=NORMAL` on a single host; `busy_timeout=5000` contention posture pinned; single-writer ceiling |
| kv | postgres | `postgres` | pooled sessions over one server; per-session `synchronous_commit` knob; the fleet state's host |
| kv | mem | `mem` | in-process ephemeral — test/reference contract anchor, never production |
| large | local | *(ungated)* | flat sharded files; fsync-durable no-replace publish (ADR-015) |
| large | pg-lo | `pg-lo` | postgres Large Objects; windowed reads; one large-object consumer per database (ADR-016) |

Engine selection has no benchmark claims beyond the recorded POC
numbers' posture in `docs/architecture/backends-and-dispatch.md` and
`docs/research/`: pick by deployment shape (single host / shared
server / ephemeral, small regime / large regime / consolidation on one
pg instance).

## Documentation

- [Architecture docs](docs/architecture/) — the authoritative record:
  the facade contract (`store-api.md`), the engine/dispatch tables, and
  ADRs 001-020.
- [CHANGELOG.md](CHANGELOG.md) — the release history, Keep-a-Changelog
  form.
- [Fuzzing](fuzz/README.md) — the sidecar targets and the
  corpus-replay gate.
- License: MIT OR Apache-2.0 ([LICENSE-MIT](LICENSE-MIT),
  [LICENSE-APACHE](LICENSE-APACHE)).