datumesh-engine 0.2.0

The Datumesh engine: the value of a cell as of a time, from stored receipts or by evaluating functions; memoised, and signed for computed cells.
Documentation

datumesh-engine

The Datumesh engine: the value of a cell as of a time. Engine::get answers with a receipt: for a stored cell, the publisher's receipt from the store, unchanged; for a computed cell, one the engine signs with the mesh's key after evaluating the cell's function through a Runtime, from the receipts of the cells it reads. Answers are memoised, computed versions are kept in the store and validated before reuse, and evaluation runs on an explicit stack, so a function may read itself (SecDB-style recursion) to any depth the limit allows. It implements spec-0.3.0 on datumesh-store 0.3.0 and datumesh-core 0.3.3. Function languages plug in through Runtime; HTTP is not part of it.

Example

[dependencies]
datumesh-engine = "0.2.0"
datumesh-store = "0.3.0"
datumesh-core = { version = "0.3.3", features = ["sign"] }

A publisher's two stored functions (net price and tax per item), a computed function adding them in a toy language sum, its runtime, and the engine answering the computed cell; the receipt it signs then verifies at level recomputed with the engine as the resolver:

use std::sync::Arc;

use datumesh_core::json::Value;
use datumesh_core::{verify, Arg, Args, Cell, CellRef, Descriptor, Keys, Level, ReceiptBuilder, Resolver, Scalar, SigningKey, Type};
use datumesh_engine::{Answer, Budget, Engine, EvalError, Inputs, Runtime};
use datumesh_store::{migrate, PgStore, Store};

/// The language `sum`: the sum of the function's inputs, each read with the cell's arguments.
struct Sum;

impl Runtime for Sum {
    fn lang(&self) -> &str {
        "sum"
    }
    fn evaluate(&self, d: &Descriptor, cell: &Cell, inputs: &mut dyn Inputs, budget: &mut Budget) -> Result<Scalar, EvalError> {
        let mut total: i128 = 0;
        for name in &d.inputs {
            budget.charge(1)?;
            let value = inputs.read(name, &cell.args)?; // propagate every read error
            total += value.lexical().parse::<i128>().map_err(|e| EvalError::Runtime(e.to_string()))?;
        }
        Scalar::from_lexical(Type::Number, &total.to_string()).map_err(|e| EvalError::Runtime(e.to_string()))
    }
}

/// A key document publishing `key` for `host`, as https://<host>/.well-known/datumesh.json serves it.
fn key_doc(host: &str, key: &SigningKey) -> String {
    let public: String = key.public_key().iter().map(|b| format!("{b:02x}")).collect();
    format!(r#"{{"host":"{host}","keys":[{{"alg":"ed25519","id":"{}","public":"{public}","valid_from":"2026-01-01T00:00:00.000000Z"}}],"proto":"datumesh/1"}}"#, key.key_id())
}

fn main() {
    let url = std::env::var("DATABASE_URL").expect("DATABASE_URL");
    let store = Arc::new(PgStore::connect(&url).unwrap());
    migrate::apply(&*store).unwrap();

    // The store verifies every receipt against its signer host's key document.
    let publisher = SigningKey::from_seed(&[1; 32]);
    let mesh = SigningKey::from_seed(&[2; 32]);
    let mut keys = Keys::new();
    for (host, key) in [("prices.example.org", &publisher), ("mesh.example.com", &mesh)] {
        store.put_namespace(host, key_doc(host, key).as_bytes(), "2026-01-01T00:00:00.000000Z").unwrap();
        keys.insert_response(host, 200, Some(key_doc(host, key).as_bytes()));
    }

    // Functions: two stored, one computed; names bound once (re-running finds them bound).
    let signature = r#"{"args":{"item":{"type":"string","valid":false}},"returns":{"type":"number","unit":"EUR"}}"#;
    let functions = [
        ("prices.example.org/net", r#"{"body":"","lang":"stored"}"#, "[]"),
        ("prices.example.org/tax", r#"{"body":"","lang":"stored"}"#, "[]"),
        ("shop.example.com/total", r#"{"body":"","lang":"sum"}"#, r#"["prices.example.org/net","prices.example.org/tax"]"#),
    ];
    for (name, code, inputs) in functions {
        let d = format!(r#"{{"code":{code},"inputs":{inputs},"signature":{signature}}}"#);
        let fn_hash = store.put_descriptor(d.as_bytes()).unwrap();
        if store.resolve_name(name, "2026-01-01T00:00:00.000000Z").unwrap().is_none() {
            store.bind_name(name, &fn_hash, "2026-01-01T00:00:00.000000Z").unwrap();
        }
    }

    // The publisher's versions: net 100 recorded on 1 March, tax 20 on 2 March.
    let item: Args = [("item".to_string(), Arg::Value(Scalar::from_lexical(Type::String, "bolt").unwrap()))].into();
    for (f, value, tx) in [("net", "100", "2026-03-01T09:00:00.000000Z"), ("tax", "20", "2026-03-02T09:00:00.000000Z")] {
        let fn_hash = store.resolve_name(&format!("prices.example.org/{f}"), tx).unwrap().unwrap();
        let cell = Cell { ns: "prices.example.org".into(), fn_name: f.into(), fn_hash, args: item.clone() };
        let receipt = ReceiptBuilder::new()
            .cell(cell.clone())
            .value(Scalar::from_lexical(Type::Number, value).unwrap())
            .valid(Args::new())
            .asof(tx)
            .tx(tx)
            .issued(tx)
            .publisher("prices.example.org")
            .signer_host("prices.example.org")
            .licence("CC-BY-4.0")
            .summary(format!("The {f} price of a bolt."))
            .sign(&publisher)
            .unwrap();
        let mut members = vec![("sig".to_string(), Value::String(receipt.sig().to_string()))];
        members.extend(datumesh_core::json::parse(receipt.payload()).unwrap().as_object().unwrap().iter().cloned());
        let hash = store.put_receipt(&Value::Object(members).to_jcs()).unwrap();
        if store.latest(&cell.id()).unwrap().is_none() {
            store.put_version(&cell.id(), tx, &hash).unwrap();
        }
    }

    // The engine: the store, the mesh's host and key, the runtimes.
    let shared: Arc<dyn Store + Send + Sync> = store.clone();
    let engine = Engine::builder(shared, "mesh.example.com").signing_key(mesh).runtime(Arc::new(Sum)).build().unwrap();
    let total = CellRef { ns: "shop.example.com".into(), fn_name: "total".into(), args: item, asof: None };
    let Answer::Receipt(bytes) = engine.get(&total, Some("2026-03-05T00:00:00.000000Z")).unwrap() else { panic!("not covered") };

    let result = verify(&bytes, &keys, Some(&engine as &dyn Resolver), None);
    assert_eq!(result.level, Some(Level::Recomputed), "{:?}", result.error());
    let receipt = datumesh_core::Receipt::from_json(&bytes).unwrap();
    println!("total = {} as of {}, tx {}", receipt.value().lexical(), receipt.asof(), receipt.tx());
    println!("verified {} receipt {} by {}", result.level.unwrap().as_str(), receipt.hash(), receipt.computed_by());
}

It prints total = 120 as of 2026-03-02T09:00:00.000000Z, tx 2026-03-02T09:00:00.000000Z, then the receipt's hash and mesh.example.com#<key id>. Run again, it validates the stored version instead of computing a new one, and prints the same.

What it does

  • Answering as of A. A receipt answers as of A only when its tx and its asof are both at or before A. Stored cell: stored_asof(cell, A), the publisher's bytes unchanged. Computed cell: the computed version with the greatest asof <= A, validated as of A; when none validates, the function is evaluated and a receipt signed (or the version kept, when the value is the same). No asof: the clock reading at the start of the request (also the issued of what it signs). Names resolve as of A, the function's and its inputs'. get_cell takes a cell with its function hash pinned.
  • Computed receipts. computed_by = <mesh host>#<key id>; inputs = the receipts read; tx = the latest input tx; asof = the latest of its input asofs and the start of every name binding it relied on (the names read, transitively; from resolve_binding), so that it holds, and recomputes, as of its own asof (an input kept across a change of its own inputs counts from where its value is known again). One receipt per version, keyed on (cell, tx, asof), stored with put_computed, never in versions; historical requests add older versions (backfill). licence and summary come from the engine's configuration.
  • Receipt identity. Two valid receipts for the same value may exist with different lineage (an input that changes and changes back: early cutoff can accept either version). After a backdated correction (an input superseded at a tx earlier than the receipt's tx), the old receipt stays signature-valid and answers the asofs it was issued for, but no longer recomputes: it is historical, not wrong. The crate documentation has the details.
  • Memo and early cutoff. A version is valid as of A when each input gives the same receipt or the same value. The memo keeps answers with the interval they hold for; stored_version_added (call it when a stored version is added) drops the answers that rest on the cell, and the unchanged ones are rechecked from memory. The memo is an optimisation: losing it (a restart) costs work, not correctness; early-cutoff records need not be persisted.
  • Recursion. An explicit stack; a read of an input not computed yet suspends the function (EvalError::Suspended), which is run again once the input is answered. Cycle, a depth limit (default 10,000; a 100,000-deep chain runs on a 2 MiB thread under a raised one).
  • Budget. Fuel per request (runtime steps plus input reads, default 10,000,000); BudgetExceeded stores nothing.
  • Single-flight. Evaluations run one at a time per engine: concurrent requests for one uncomputed cell evaluate it once. Memoised and stored answers do not wait.
  • Resolver. Engine is a time-aware datumesh_core::Resolver with the same answering rule; computed cells are evaluated, never read back from their own receipts. The core's test kit passes every check under its default answering rule TxAndAsof (tests/conformance.rs).

The crate documentation describes the Runtime and Inputs contract for function languages and the get contract for an HTTP layer (answers, errors, envelope outcomes). The API is synchronous; Engine is Send + Sync. Dependencies: datumesh-core (feature sign), datumesh-store, and tracing (failures that Resolver::evaluate turns into None, and computed-version conflicts, at target datumesh_engine). No unsafe, no panics on any input, no global state, no network, no file but the configured key file.

Feature native-test: native_test::NativeTest, a runtime of Rust closures keyed by function hash, for tests and the benchmark.

Development

With a PostgreSQL 16 server whose role may create databases in DATABASE_URL, cargo test --all-features runs everything (each test in a database of its own, all on PgStore; the 100,000-deep chain of tests/recursion.rs takes about 8 minutes of it). tools/cargo.sh <toolchain> <args> runs cargo in a rust:<toolchain> container next to a datumesh-pg container; tools/package_test.sh is the packaging test; tools/public_api.sh regenerates public-api.txt and public-api-all-features.txt (cargo public-api); tools/crosscheck.sh verifies every receipt the test suite signs with the Python and JavaScript verifiers (datumesh 0.3.0, @datumesh/verify 0.3.0) at level signed.

Benchmark

examples/bench.rs (cargo run --release --features native-test --example bench), against PostgreSQL 16 with default settings on the development host (6 vCPU, 15 GB RAM, synchronous_commit on), one call at a time; the full run is in bench/results/2026-10-24-0.2.0.md (0.1.0's in bench/results/2026-10-24.md):

Operation p50 p99
warm get stored, by reference (resolving the name: 1 query) 173 µs 371 µs
warm get_cell stored (pinned: no query) 5.3 µs 16 µs
warm get computed, by reference (2 name queries) 373 µs 757 µs
warm get_cell computed (1 name query) 182 µs 367 µs
cold get computed, first ever (sign, put_receipt, put_computed) 5.5 ms 12.9 ms
get computed, empty memo, version in the store (validated) 1.6 ms 3.6 ms
cold get of a diamond (4 computed cells) 20 ms 26 ms
cold get of a 1,000-deep chain 4.53 s 4.63 s
signing one receipt 274 µs 494 µs
storing one receipt (put_receipt verifies it) 2.8 ms 4.2 ms

A 10,000-cell graph (100 layers of 100, each cell reading two below): built cold in 50 s; one correction of a stored input under it, until the next get of the top is correct: 27 s when the change propagates (about 5,000 cells recomputed, signed and stored), 212 ms when early cutoff stops it at the first layer (the rest rechecked from memory). The memo then holds 10,101 answers in about 27 MB (2.7 KB each, the receipt's bytes included); the dirty set is empty at rest (it keeps marks only while an evaluation that began before them runs).

Licensed under Apache-2.0.