datumesh-store

A store for the Datumesh protocol, on PostgreSQL or (since 0.4.0) in one SQLite file: function descriptors, signed receipts, blobs, versions of cells over transaction time (stored and computed), lineage (slices and their manifests included), the members of slices, namespaces' key documents and function names. PgStore and SqliteStore implement the Store trait and, through it, datumesh-core's Resolver, so datumesh_core::verify recomputes receipts of stored cells from the database. One conformance suite runs against both backends. Each passes datumesh-core's resolver test kit as a time-aware resolver under the kit's default answer rule (AnswerRule::TxAndAsof: a version answers as of A when its tx and its receipt's asof are both at or before A), with the slice checks (Kit::slices(true): it serves manifests). Store is sealed: only this crate implements it (see The store).

• datumesh-store: the library.
• datumesh-store-cli: the datumesh-store command: migrate [--check], import, export, bench.

Example

[dependencies]
datumesh-store = "0.4.0"
datumesh-core = "0.3.4"

Store the key documents of spec/vectors/receipt/stored-number.json, its receipt (verified against them on the way in), its function's descriptor and the version it states, then verify the receipt through the store (it recomputes the stored value as of the receipt's asof):

use datumesh_core::json::{self, Value};
use datumesh_core::{verify, Keys, Level, Receipt, Resolver};
use datumesh_store::{migrate, PgStore, Store};

fn main() {
    let url = std::env::var("DATABASE_URL").expect("DATABASE_URL");
    // A store connects to an empty database too; the schema comes through it (the
    // `datumesh-store migrate` command does the same). Re-running is a no-op.
    let store = PgStore::connect(&url).unwrap();
    migrate::apply(&store).unwrap();

    let vector = json::parse(&std::fs::read("stored-number.json").unwrap()).unwrap();
    let text = vector.get("input").and_then(Value::as_str).unwrap().as_bytes();
    let descriptor = vector.get("recompute").and_then(|r| r.get("descriptor")).unwrap().to_jcs();

    // Receipts are verified on the way in, against the key document stored for their
    // signer host: store the host's document first. The vector carries the documents.
    let mut keys = Keys::new();
    for (host, env) in vector.get("keys").and_then(Value::as_object).unwrap() {
        let body = env.get("https").and_then(|h| h.get("body")).and_then(Value::as_str).unwrap();
        store.put_namespace(host, body.as_bytes(), "2024-03-16T09:00:00.000000Z").unwrap();
        keys.insert_response(host, 200, Some(body.as_bytes()));
    }

    let fn_hash = store.put_descriptor(&descriptor).unwrap();
    let hash = store.put_receipt(text).unwrap();
    let receipt = Receipt::from_json(text).unwrap();
    let cell_id = receipt.cell().id();
    // Versions are append-only: on a second run the version is already there.
    if store.latest(&cell_id).unwrap().is_none() {
        store.put_version(&cell_id, receipt.tx(), &hash).unwrap();
    }

    let result = verify(text, &keys, Some(&store as &dyn Resolver), None);
    assert_eq!(result.level, Some(Level::Recomputed), "{:?}", result.error());
    println!("verified {} receipt {}", result.level.unwrap().as_str(), hash);
    println!("fx_rate is {fn_hash}; the cell's latest version is at {}", store.latest(&cell_id).unwrap().unwrap().tx);
}

It prints verified recomputed receipt 19a5fe3a744c5792f747fb4c286ce396ab44b95922de4f07937502fe595b6f81, then the function hash and the version's tx.

The same in one SQLite file (no server; SQLite is compiled in, feature sqlite, on by default):

use datumesh_core::json::{self, Value};
use datumesh_core::{verify, Keys, Level, Receipt, Resolver};
use datumesh_store::{migrate, SqliteStore, Store};

fn main() {
    // A path (created if missing), or a sqlite: URL with SqliteStore::connect.
    let path = std::env::var("STORE_PATH").unwrap_or_else(|_| "store.db".into());
    let store = SqliteStore::open(&path).unwrap();
    migrate::sqlite::apply(&store).unwrap();

    let vector = json::parse(&std::fs::read("stored-number.json").unwrap()).unwrap();
    let text = vector.get("input").and_then(Value::as_str).unwrap().as_bytes();
    let descriptor = vector.get("recompute").and_then(|r| r.get("descriptor")).unwrap().to_jcs();
    let mut keys = Keys::new();
    for (host, env) in vector.get("keys").and_then(Value::as_object).unwrap() {
        let body = env.get("https").and_then(|h| h.get("body")).and_then(Value::as_str).unwrap();
        store.put_namespace(host, body.as_bytes(), "2024-03-16T09:00:00.000000Z").unwrap();
        keys.insert_response(host, 200, Some(body.as_bytes()));
    }
    store.put_descriptor(&descriptor).unwrap();
    let hash = store.put_receipt(text).unwrap();
    let receipt = Receipt::from_json(text).unwrap();
    let cell_id = receipt.cell().id();
    if store.latest(&cell_id).unwrap().is_none() {
        store.put_version(&cell_id, receipt.tx(), &hash).unwrap();
    }
    let result = verify(text, &keys, Some(&store as &dyn Resolver), None);
    assert_eq!(result.level, Some(Level::Recomputed), "{:?}", result.error());
    println!("verified {} receipt {} in {path}", result.level.unwrap().as_str(), hash);
}

The API is synchronous. From async code, call it on a blocking thread (tokio::task::spawn_blocking). PgStore is Send + Sync; share one per process.

The store

Every Store method returns a Result. Malformed input is refused with the core's error code (Error::Invalid), without a query: hashes and cell ids are 64 lowercase hex digits, instants YYYY-MM-DDTHH:MM:SS.ffffffZ, hosts DNS names, function names fully qualified. A database failure is Error::Database (or Error::PoolTimeout), holding a DatabaseError: message() (the server's message, or the driver's text), code() (the SQLSTATE, e.g. 42P01), and the driver's error as its source(); reads answer Ok(None) / Ok(vec![]) only when nothing is stored. The Resolver methods (descriptor, receipt, manifest, evaluate) keep the core's Option signatures: a malformed argument or a database failure is None there (a failure is logged with tracing, target datumesh_store), which verify reports as unavailable.

Store is sealed (a private supertrait): only this crate implements it. The store is the trust boundary: every receipt in it verified against its host's key document when it was stored, and callers rely on what it answers as checked; an implementation elsewhere could answer anything. It is still a trait object (Arc<dyn Store + Send + Sync>).

PgStore::connect(url) and PgStore::connect_with(url, PoolOptions) connect (TLS: see below). Connecting opens one connection and does nothing else on it (no statement is prepared, the schema is not looked at), so a store on an empty database is how the schema is made: migrate::apply(&store). The pool is the crate's own: a mutex and a condition variable, no global state, no background threads, a bounded number of connections, a checkout timeout, connections the server closed are replaced, and each statement is prepared once per connection. Every put_* is one transaction.

TLS

connect and connect_with always use a rustls connector (tokio-postgres-rustls, the ring provider chosen for that connector; no process-wide crypto provider is installed). Whether TLS is used is the URL's sslmode: disable never, prefer (the default) when the server offers it, require always. Whenever TLS is used, the server's certificate and host name are verified, against the Mozilla roots (webpki-roots, compiled in) plus any roots added through PoolOptions; there is no option to accept any certificate.

let pem = std::fs::read("ca.pem")?; // one or more certificates
let options = PoolOptions::new().tls_root_certificates_pem(&pem)?; // appended to the Mozilla roots
let store = PgStore::connect_with("postgres://app@db.internal/datumesh?sslmode=require", options)?;

PoolOptions::tls_webpki_roots(false) trusts only the added roots (a private CA). A bundle with no certificate, or one that does not parse, is Error::Config. A certificate that does not verify is Error::Database from connect, its message naming the reason (invalid peer certificate: UnknownIssuer, certificate not valid for name ...).

So a server with ssl=on and a self-signed certificate (or one from a private CA) fails even under the default sslmode=prefer: prefer does not fall back to plain text when the certificate does not verify. Add its CA with tls_root_certificates_pem, or connect with sslmode=disable. Client certificates are not configurable through PoolOptions; use pg_unstable::with_connector for them.

pg_unstable

No type of the postgres driver or of a TLS crate is part of the API, except in the module pg_unstable, which is outside semver until 1.0: it exposes the postgres driver's types, and any release may change it, including a patch release. It holds with_connector(f, PoolOptions) (a store whose connections come from any function returning a postgres::Client: client certificates, another TLS setup, session settings), Connector (plain postgres clients that connect exactly as the store does) and pub use postgres (the driver this crate is built with). Depend on an exact version (datumesh-store = "=0.4.0") if you use it.

Schema

Plain SQL in migrations/ (migrations at the top is a link to it), embedded in the library, applied in order with a tracking table (datumesh_schema_migrations, with checksums) under an advisory lock.

• descriptors(fn_hash, bytes), receipts(hash, bytes, cell_id, tx, tx_at, fn_hash, publisher, asof, asof_at), blobs(sha256, bytes, length, media_type), versions(cell_id, tx, tx_at, receipt_hash), computed_versions(cell_id, tx, tx_at, asof, asof_at, receipt_hash) (primary key (cell_id, tx, asof)), edges(receipt_hash, input_hash, kind) (kind is receipt), namespaces(host, keydoc, validated_at, validated_at_at, suspended_at, suspended_at_at), name_bindings(fn_name, tx, tx_at, fn_hash).
• receipts.asof (migration 2) is the receipt's own asof: backfilled from the stored canonical bytes when the migration runs, checked equal to them (a CHECK), and taken from them by a trigger when an insert does not name it (a 0.1.x writer).
• Migration 3 (0.3.1): cells(cell_id, fn_hash, ns, fn, nargs) and cell_args(cell_id, name, type, value, num, fn_hash), the typed-argument index: every cell of a stored receipt, each argument's type and lexical form and, for a number, its exact numeric value, indexed by (function, argument, value) and (function, argument, number). A trigger on receipts derives the rows from the canonical bytes (so a 0.3.0 writer's receipts are indexed too), and the migration backfills them the same way. manifests(entry, slice_id, members_hash, fn_hash, members, bytes), manifest_members(entry, member_hash) (indexed by member), slice_edges(receipt_hash, entry) (indexed by entry). All five are insert-only.
• Migration 4 (0.3.2): receipts.value_type and receipts.value, the receipt's typed value (backfilled from the canonical bytes, checked equal to them, filled by a trigger when an insert does not name them), and the indexes receipts (fn_hash, asof) and receipts (cell_id, asof).
• Hashes, ids, names and instants are text COLLATE "C": instants (fixed width) compare chronologically as text, and every as-of query compares the text. Each instant is also timestamptz (*_at), checked equal to the text; it round-trips every instant from 0001 to 9999 exactly, whatever the session's TimeZone or DateStyle (tested).
• computed_versions is insert-only in the database (BEFORE UPDATE OR DELETE and BEFORE TRUNCATE triggers raise) but in any order: there is no tx-order rule; its primary key serialises concurrent writers of one key.
• versions and name_bindings are append-only in the database: BEFORE UPDATE OR DELETE row triggers and BEFORE TRUNCATE statement triggers raise; a BEFORE INSERT trigger takes a per-cell (per-name) transaction-scoped advisory lock and refuses a tx not later than the latest, so concurrent writers, raw SQL included, are serialised. It requires READ COMMITTED (the store's sessions use it). A superuser can still disable triggers or edit the catalog; the triggers protect against mistakes and ordinary roles, not against the database owner.

datumesh-store migrate --check migrates a scratch database (the role needs CREATEDB; migrate::check(&live, &scratch) takes a store on each), compares the two catalogs (relations, columns, constraints, indexes, triggers and their enabled state, functions, recorded migrations), prints the differences and exits 1 on drift; then it prepares every statement the store uses. The sync driver has no compile-time query checking, so the test suite prepares every statement against the migrated schema.

SQLite (0.4.0)

SqliteStore::open(path) (or open_with(path, SqliteOptions), or connect("sqlite:path")) stores everything in one file, creating it when it is missing (not its directory); SqliteStore::open_existing(path) opens only an existing file and changes nothing about it (for checks: migrate::sqlite::check_file(path)), with SQLite compiled into the crate (rusqlite with bundled: no system library, the same SQLite on Linux, macOS and Windows). Same methods, errors and rules as PgStore; one conformance suite runs every shared test against both. Its schema is one migration, migrations/sqlite/0001_initial.sql (migrate::sqlite::apply, catalog, check; check migrates a scratch database in memory). The differences from the PostgreSQL schema:

• STRICT tables; TEXT for hashes, ids, names and instants (SQLite's default collation is byte order, as COLLATE "C"), BLOB for bytes; no timestamptz (*_at) columns; GLOB checks instead of regular expressions.
• cell_args.sort_key instead of cell_args.num numeric: a text that sorts as the exact decimal does (sign class; the decimal exponent plus 500 in three digits; the significant digits; for a negative number exponent and digits complemented and ~ appended, so prefixes reverse too). Every other type's key is its lexical form. One index (fn_hash, name, type, sort_key, cell_id) serves all ranges. Tested on the hard cases: 9 < 10, -1.5 < -1.25 < -1, 0.1 < 0.109 < 0.11, zero, and 128-character numbers at both ends and next to zero.
• The derived columns (receipts.asof, value_type, value) and the typed-argument index are written by the store in the receipt's transaction (there is no older SQLite writer to serve); the receipt columns are still checked against the bytes (json_extract).
• Append-only by triggers on the same tables (no TRUNCATE in SQLite; DELETE without WHERE fires the delete triggers), also on receipts, edges and descriptors (SQLite enforces foreign keys only on connections that ask, so referenced rows protect themselves), plus triggers refusing an insert over an existing key (so INSERT OR REPLACE cannot replace a row) and an insert naming a receipt, descriptor, cell or manifest that is not stored.
• receipts_fn_hash is not created (receipts_fn_hash_asof covers it).

Concurrency: the file is in WAL mode (readers never wait); every put_* is one BEGIN IMMEDIATE transaction, which takes the database's single write lock before its checks, so check-then-insert (NotLater, Conflict, idempotent puts) is atomic against every other writer, thread or process; the busy timeout (SqliteOptions::busy_timeout, 30 s) makes writers wait rather than fail, and a store's own threads queue on a mutex first (SQLite's busy handler polls with sleeps). The append-only and order rules are triggers, so a raw connection is held to them as well. tests/sqlite_processes.rs runs four processes of four threads each (plus a raw-SQL writer each) against one file and checks that the history is in tx order by insertion, that every failure is NotLater or Conflict, and that one receipt and one manifest put by everyone are stored once. A commit that finds the write-ahead log beyond 64 MiB checkpoints and truncates it (under a steady load of readers SQLite's automatic checkpoints never restart the log). Commits are durable (synchronous = FULL) unless SqliteOptions::durable(false) (NORMAL: a power loss can lose the latest commits, never consistency). A file on a network file system is not supported.

The command

export DATABASE_URL=postgres://postgres:postgres@localhost:5432/postgres   # ?sslmode=require for TLS only
export DATABASE_CA_FILE=ca.pem         # optional: roots to trust besides the Mozilla roots
datumesh-store migrate                 # apply; re-running is a no-op
datumesh-store migrate --check         # exit 1 on drift
datumesh-store import spec/vectors     # vector files or directories; JSON lines; - for stdin
datumesh-store export > store.jsonl    # JSON lines: namespaces, descriptors, receipts, manifests, blobs, versions, computed versions, names
datumesh-store bench                   # see below
DATABASE_URL=store.db datumesh-store migrate   # a SQLite file (or sqlite:store.db, sqlite:///abs/path)

DATABASE_URL names PostgreSQL when it is a postgres:// / postgresql:// URL or a key=value connection string, and a SQLite file otherwise (a sqlite: URL or a path; created if missing, except by migrate --check, which fails with no such database file and creates nothing). An export from one backend imports into the other, and both export the same lines for the same contents. Every PostgreSQL command connects as PgStore::connect_with does (TLS as the URL's sslmode says, verified against the Mozilla roots plus DATABASE_CA_FILE). export writes one consistent snapshot in an order import replays (namespaces first: receipts are verified against them); importing what is already there changes nothing; exports of 0.1.x (without computed versions), 0.2.x and 0.3.0 (without manifests) import too. A store keeps one key document per host, the last validated, and an export carries only that one: receipts verified under a document that was replaced since cannot be imported from the export again (E_KEY_UNKNOWN). The specification's vectors are such a set since spec-0.4.0 (its slice-lineage receipt vectors sign for three hosts with other keys than the earlier vectors). Importing vectors stores each valid receipt vector's key documents (validated at the time of the import, unless the same document is already stored) before its receipts, with its recompute descriptor, inputs and (since 0.3.1) manifests, and each valid slice vector that is a manifest.

Development

docker compose up -d starts PostgreSQL 16; with DATABASE_URL set, cargo test --workspace runs everything (each test in a database of its own, so the role needs CREATEDB). tools/cargo.sh <toolchain> <args> runs cargo in a rust:<toolchain> container next to a datumesh-pg container; tools/package_test.sh is the packaging test; tools/bench.sh the benchmark. The TLS tests need a server with TLS: tools/tls_pg.sh makes a throwaway CA and server certificate (in target-tls/, never committed) and starts the container datumesh-tls-pg; then set DATABASE_TLS_URL=postgres://postgres:postgres@datumesh-tls-pg:5432/postgres and DATABASE_TLS_CA_FILE=/w/target-tls/ca.crt (without them the TLS tests are skipped; CI sets DATUMESH_REQUIRE_TLS_TESTS, which makes them fail instead). tools/public_api.sh regenerates public-api.txt (cargo public-api) and checks that no driver or TLS type appears outside pg_unstable. The specification (spec/, a submodule at spec-0.4.0) provides the vectors.

Benchmark

tools/bench.sh runs datumesh-store bench (release build) against a dedicated postgres:16 container whose data directory is a host volume, then times a cold restart. Results on the development host (6 vCPU, 15 GB RAM, virtual disk; 2026-10-21), 1,000,000 versions over 10,000 cells (100 each), every version a receipt signed with feature sign, verified by put_receipt against the stored key document and stored with put_version, from 6 threads: 660.2 s, 1,515 versions/s (loading sessions with synchronous_commit = off; before put_receipt verified signatures the same load took 546.9 s, 1,829 versions/s). Plus 1,000 receipts depending on one input. Timings are one call at a time through PgStore, synchronous_commit = on, 10,000 samples:

Disk: receipts 1,129 MB + 324 MB indexes, versions 197 MB + 314 MB indexes; 1,965 MB for all tables (about 2.0 KB per version, of which the canonical receipt is about 1.1 KB). Cold restart: docker stop 0.4 s (about 4 s right after the load, a shutdown checkpoint); after dropping the page cache, docker start to the first answered version_asof 1.2-1.6 s, of which about 0.5 s is starting the tool's own container; the probe itself answered 112-168 ms after it started.

Version 0.2.0's additions, at 1,000 cells × 100 versions on the development server with default settings (warm cache; bench/results/2026-10-24), p50 / p99: computed_asof 221 / 505 µs, stored_asof 226 / 476 µs (version_asof in the same run 238 / 534 µs), computed_asof_with_receipt 222 / 513 µs and stored_asof_with_receipt 219 / 516 µs (against 397 / 744 and 387 / 727 µs for the read followed by receipt), put_computed 1.72 / 2.69 ms, cell_dependents with 1,000 dependent cells 6.38 / 11.29 ms. Version 0.3.0's resolve_binding in the same setting: 188 / 464 µs (resolve_name 169 / 362 µs), 100 names × 100 bindings.

Version 0.3.1's slice operations in the same setting (bench/results/2026-10-25, --slice-cells 100000), p50 / p99: slice_members of 1,000 members 67.5 / 156 ms (a range of 1,000 of 100,000 cells) and 53.2 / 85.1 ms (every cell of a function with 1,000), of 100,000 members 5.68 / 6.36 s (about 57 µs per member, receipts included); versions_between 0.51 / 1.86 ms; put_manifest of 1,000 members 62.1 / 89.7 ms, of 100,000 5.98 / 7.01 s; manifest of 1,000 members 0.28 / 0.93 ms, of 100,000 14.2 / 33.1 ms.

Blobs (put_blob / blob, median, two runs): 1 MiB 34 / 5-6 ms, 4 MiB 127-135 / 19 ms, 16 MiB 330-473 / 63-82 ms, 64 MiB 1,683-1,875 / 298-310 ms; writes level off near 30-50 MB/s from 1 MiB on and every value is read and written whole, in memory on both sides. Keep blobs up to 1 MiB in the table; above that, object storage (content-addressed by the same SHA-256) is the better home.

Server settings (tools/bench.sh): shared_buffers=4GB, effective_cache_size=11GB, work_mem=32MB, maintenance_work_mem=1GB, wal_buffers=16MB, max_wal_size=8GB, min_wal_size=1GB, checkpoint_completion_target=0.9, wal_compression=lz4, random_page_cost=1.1, effective_io_concurrency=200, max_worker_processes=6, max_parallel_workers=6, max_parallel_workers_per_gather=3, max_parallel_maintenance_workers=3, max_connections=50, jit=off, --shm-size=1g; fsync and synchronous_commit on.

Licensed under Apache-2.0.