datumesh-engine
The Datumesh engine: the value of a cell as of a time. Engine::get answers with a receipt: for a stored cell, the publisher's receipt from the store, unchanged; for a computed cell, one the engine signs with the mesh's key after evaluating the cell's function through a Runtime, from the receipts of the cells it reads. Answers are memoised, computed versions are kept in the store and validated before reuse, and evaluation runs on an explicit stack, so a function may read itself (SecDB-style recursion) to any depth the limit allows. It implements spec-0.3.0 on datumesh-store 0.2.0 and datumesh-core 0.3.2. Function languages plug in through Runtime; HTTP is not part of it.
Example
[]
= "0.1.0"
= "0.2.0"
= { = "0.3.2", = ["sign"] }
A publisher's two stored functions (net price and tax per item), a computed function adding them in a toy language sum, its runtime, and the engine answering the computed cell; the receipt it signs then verifies at level recomputed with the engine as the resolver:
use Arc;
use Value;
use ;
use ;
use ;
/// The language `sum`: the sum of the function's inputs, each read with the cell's arguments.
;
/// A key document publishing `key` for `host`, as https://<host>/.well-known/datumesh.json serves it.
It prints total = 120 as of 2026-03-02T09:00:00.000000Z, tx 2026-03-02T09:00:00.000000Z, then the receipt's hash and mesh.example.com#<key id>. Run again, it validates the stored version instead of computing a new one, and prints the same.
What it does
- Answering as of A. A receipt answers as of A only when its
txand itsasofare both at or before A. Stored cell:stored_asof(cell, A), the publisher's bytes unchanged. Computed cell: the computed version with the greatestasof <= A, validated as of A; when none validates, the function is evaluated and a receipt signed (or the version kept, when the value is the same). No asof: the clock reading at the start of the request (also theissuedof what it signs). Names resolve as of A, the function's and its inputs'.get_celltakes a cell with its function hash pinned. - Computed receipts.
computed_by=<mesh host>#<key id>;inputs= the receipts read;tx= the latest inputtx;asof= the latest inputasof(SPEC 8.2 rule 2), later only where needed for the receipt to hold as of its ownasof(an input kept across a change of its own inputs; a name bound after it). One receipt per version, keyed on (cell,tx,asof), stored withput_computed, never inversions; historical requests add older versions (backfill).licenceandsummarycome from the engine's configuration. - Receipt identity. Two valid receipts for the same value may exist with different lineage (an input that changes and changes back: early cutoff can accept either version). After a backdated correction (an input superseded at a
txearlier than the receipt'stx), the old receipt stays signature-valid and answers the asofs it was issued for, but no longer recomputes: it is historical, not wrong. The crate documentation has the details. - Memo and early cutoff. A version is valid as of A when each input gives the same receipt or the same value. The memo keeps answers with the interval they hold for;
stored_version_added(call it when a stored version is added) drops the answers that rest on the cell, and the unchanged ones are rechecked from memory. The memo is an optimisation: losing it (a restart) costs work, not correctness; early-cutoff records need not be persisted. - Recursion. An explicit stack; a read of an input not computed yet suspends the function (
EvalError::Suspended), which is run again once the input is answered.Cycle, a depth limit (default 10,000; a 100,000-deep chain runs on a 2 MiB thread under a raised one). - Budget. Fuel per request (runtime steps plus input reads, default 10,000,000);
BudgetExceededstores nothing. - Single-flight. Evaluations run one at a time per engine: concurrent requests for one uncomputed cell evaluate it once. Memoised and stored answers do not wait.
- Resolver.
Engineis a time-awaredatumesh_core::Resolverwith the same answering rule; computed cells are evaluated, never read back from their own receipts. The core's test kit passes except at the one point where its tx-only rule differs (a late-signed version as of its tx), astests/conformance.rspins.
The crate documentation describes the Runtime and Inputs contract for function languages and the get contract for an HTTP layer (answers, errors, envelope outcomes). The API is synchronous; Engine is Send + Sync. Dependencies: datumesh-core (feature sign), datumesh-store, and tracing (failures that Resolver::evaluate turns into None, and computed-version conflicts, at target datumesh_engine). No unsafe, no panics on any input, no global state, no network, no file but the configured key file.
Feature native-test: native_test::NativeTest, a runtime of Rust closures keyed by function hash, for tests and the benchmark.
Development
With a PostgreSQL 16 server whose role may create databases in DATABASE_URL, cargo test --all-features runs everything (each test in a database of its own). tools/cargo.sh <toolchain> <args> runs cargo in a rust:<toolchain> container next to a datumesh-pg container; tools/package_test.sh is the packaging test; tools/public_api.sh regenerates public-api.txt and public-api-all-features.txt (cargo public-api); tools/crosscheck.sh verifies every receipt the test suite signs with the Python and JavaScript verifiers (datumesh 0.3.0, @datumesh/verify 0.3.0) at level signed.
Benchmark
examples/bench.rs (cargo run --release --features native-test --example bench), against PostgreSQL 16 with default settings on the development host (6 vCPU, 15 GB RAM, synchronous_commit on), one call at a time; the full run is in bench/results/2026-10-24.md:
| Operation | p50 | p99 |
|---|---|---|
warm get stored, by reference (resolving the name: 1 query) |
191 µs | 414 µs |
warm get_cell stored (pinned: no query) |
5.2 µs | 18 µs |
warm get computed, by reference (2 name queries) |
364 µs | 691 µs |
warm get_cell computed (1 name query) |
200 µs | 405 µs |
cold get computed, first ever (sign, put_receipt, put_computed) |
6.0 ms | 12.1 ms |
get computed, empty memo, version in the store (validated) |
2.0 ms | 3.4 ms |
cold get of a diamond (4 computed cells) |
27 ms | 32 ms |
cold get of a 1,000-deep chain |
5.81 s | 5.83 s |
| signing one receipt | 272 µs | 492 µs |
storing one receipt (put_receipt verifies it) |
2.6 ms | 3.8 ms |
A 10,000-cell graph (100 layers of 100, each cell reading two below): built cold in 59 s; one correction of a stored input under it, until the next get of the top is correct: 29 s when the change propagates (about 5,000 cells recomputed, signed and stored), 162 ms when early cutoff stops it at the first layer (the rest rechecked from memory). The memo then holds 10,101 answers in about 27 MB (2.7 KB each, the receipt's bytes included); the dirty set is empty at rest (it keeps marks only while an evaluation that began before them runs).
Licensed under Apache-2.0.