datumesh-engine
The Datumesh engine: the value of a cell as of a time. Engine::get answers with a receipt: for a stored cell, the publisher's receipt from the store, unchanged; for a computed cell, one the engine signs with the mesh's key after evaluating the cell's function through a Runtime, from the receipts of the cells it reads. Answers are memoised, computed versions are kept in the store and validated before reuse, and evaluation runs on an explicit stack, so a function may read itself (SecDB-style recursion) to any depth the limit allows. It implements spec-0.4.0 (slice-as-lineage, transaction-time reads, evaluation faults, normative fuel) on datumesh-store 0.3.2 and datumesh-core 0.3.4. Function languages plug in through Runtime; HTTP is not part of it.
Example
[]
= "0.3.1"
= "0.3.2"
= { = "0.3.4", = ["sign"] }
A publisher's two stored functions (net price and tax per item), a computed function adding them in a toy language sum, its runtime, and the engine answering the computed cell; the receipt it signs then verifies at level recomputed with the engine as the resolver:
use Arc;
use Value;
use ;
use ;
use ;
/// The language `sum`: the sum of the function's inputs, each read with the cell's arguments.
;
/// A key document publishing `key` for `host`, as https://<host>/.well-known/datumesh.json serves it.
It prints total = 120 as of 2026-03-02T09:00:00.000000Z, tx 2026-03-02T09:00:00.000000Z, then the receipt's hash and mesh.example.com#<key id>. Run again, it validates the stored version instead of computing a new one, and prints the same.
What it does
- Answering as of A. A receipt answers as of A only when its
txand itsasofare both at or before A. Stored cell:stored_asof(cell, A), the publisher's bytes unchanged. Computed cell: the computed version with the greatestasof <= A, validated as of A; when none validates, the function is evaluated and a receipt signed (or the version kept, when the value is the same). No asof: the clock reading at the start of the request (also theissuedof what it signs). Names resolve as of A, the function's and its inputs'.get_celltakes a cell with its function hash pinned. - Computed receipts.
computed_by=<mesh host>#<key id>;inputs= the receipts read;tx= the latest inputtx;asof= the latest of its input asofs and the start of every name binding it relied on (the names read, transitively; fromresolve_binding), so that it holds, and recomputes, as of its ownasof(an input kept across a change of its own inputs counts from where its value is known again). One receipt per version, keyed on (cell,tx,asof), stored withput_computed, never inversions; historical requests add older versions (backfill).licenceandsummarycome from the engine's configuration. - Receipt identity. Two valid receipts for the same value may exist with different lineage (an input that changes and changes back: early cutoff can accept either version). After a backdated correction (an input superseded at a
txearlier than the receipt'stx), the old receipt stays signature-valid and answers the asofs it was issued for, but no longer recomputes: it is historical, not wrong. The crate documentation has the details. - Memo and early cutoff. A version is valid as of A when each input gives the same receipt or the same value. The memo keeps answers with the interval they hold for;
stored_version_added(call it when a stored version is added) drops the answers that rest on the cell, and the unchanged ones are rechecked from memory. The memo is an optimisation: losing it (a restart) costs work, not correctness; early-cutoff records need not be persisted. - Slices (spec-0.4.0).
Inputs::read_slicereads a slice as of A in key order: a stored function's matching cells (Store::slice_members), or a computed function's declaredextent, each member a cell the engine evaluates and memoises (a member on the stack is a cycle; a member without a value makes the readE_FORMULA_ABSENT). Oneinputsentryslice:<slice id>:<members hash>per slice; its manifest is stored and served by the engine as a resolver. Validation compares members hashes, then the members' values;stored_version_addeddrops the answers whose slice pattern the new version falls into, and nothing else (a correction to one day of a running total drops that day and the later ones).read_versions(a version slice) andread_asof(a cell as of a transaction timeT,E_FORMULA_FUTUREafter A) complete the reads; a computed receipt'sasofcounts every member, binding start,Tandto. - Faults. A runtime's
EvalError::Fault(E_FORMULA_DIV_ZERO, ...) gives no receipt:Answer::NotCovered { fault: Some(code) }, for the envelope'sfault. Nothing is stored for the faulted cell; a reader gets the same fault and may catch it. - Asof-dependent functions (
Runtime::asof_dependent, or readingInputs::asof): the receipt'sasofis the evaluation asof, one receipt per requested asof, never reused for another asof by the memo or the store. - Recursion. An explicit stack; a read of an input not computed yet suspends the function (
EvalError::Suspended), which is run again once the input is answered.Cycle, a depth limit (default 10,000; a 100,000-deep chain runs on a 2 MiB thread under a raised one). - Budget. Fuel is normative (SPEC 15.12): each attempt of an evaluation has its own meter, and a cell's fuel is that of its completed attempt, whatever the restarts; the request's fuel (default 10,000,000) bounds the sum, checked at every charge against a lower bound. A work limit (default 16 times the fuel) bounds everything else, discarded attempts included.
BudgetExceededandWorkExceededstore nothing. - Single-flight. Evaluations run one at a time per engine: concurrent requests for one uncomputed cell evaluate it once. Memoised and stored answers do not wait.
- Resolver.
Engineis a time-awaredatumesh_core::Resolverwith the same answering rule; computed cells are evaluated, never read back from their own receipts, andevaluationgives the lineage (so a verifier reportsE_INPUT_SUPERSEDEDfor a receipt whose input was superseded);manifestserves stored manifests. The core's test kit passes every check under its default answering ruleTxAndAsof, with the slice checks (Kit::slices(true),tests/conformance.rs).
The crate documentation describes the Runtime and Inputs contract for function languages and the get contract for an HTTP layer (answers, errors, envelope outcomes). The API is synchronous; Engine is Send + Sync. Dependencies: datumesh-core (feature sign), datumesh-store, and tracing (failures that Resolver::evaluate turns into None, and computed-version conflicts, at target datumesh_engine). No unsafe, no panics on any input, no global state, no network, no file but the configured key file.
Feature native-test: native_test::NativeTest, a runtime of Rust closures keyed by function hash, for tests and the benchmark.
Development
With a PostgreSQL 16 server whose role may create databases in DATABASE_URL, cargo test --all-features runs everything (each test in a database of its own, all on PgStore; the 100,000-deep chain of tests/recursion.rs takes about 8 minutes of it). tools/cargo.sh <toolchain> <args> runs cargo in a rust:<toolchain> container next to a datumesh-pg container; tools/package_test.sh is the packaging test; tools/public_api.sh regenerates public-api.txt and public-api-all-features.txt (cargo public-api); tools/crosscheck.sh verifies every receipt the test suite signs with the Python and JavaScript verifiers (datumesh 0.4.0, @datumesh/verify 0.4.0) at level signed, and checks every manifest against its slice entry.
Benchmark
examples/bench.rs (cargo run --release --features native-test --example bench), against PostgreSQL 16 with default settings on the development host (6 vCPU, 15 GB RAM, synchronous_commit on), one call at a time; the full run is in bench/results/2026-10-25-0.3.0.md (0.2.0's and 0.1.0's beside it; 0.3.1's, back to back with 0.2.0 and with the early-cutoff case repeated, in bench/results/2026-10-26-0.3.1.md):
| Operation | p50 | p99 |
|---|---|---|
warm get stored, by reference (resolving the name: 1 query) |
197 µs | 401 µs |
warm get_cell stored (pinned: no query) |
5.3 µs | 15 µs |
warm get computed, by reference (2 name queries) |
375 µs | 777 µs |
warm get_cell computed (1 name query) |
217 µs | 416 µs |
cold get computed, first ever (sign, put_receipt, put_computed) |
6.6 ms | 9.2 ms |
get computed, empty memo, version in the store (validated) |
1.6 ms | 3.0 ms |
cold get of a diamond (4 computed cells) |
28 ms | 39 ms |
cold get of a 1,000-deep chain |
5.6 s | 5.8 s |
| signing one receipt | 290 µs | 547 µs |
storing one receipt (put_receipt verifies it) |
2.6 ms | 4.0 ms |
A 10,000-cell graph (100 layers of 100, each cell reading two below): built cold in 58 s; one correction of a stored input under it, until the next get of the top is correct: 29 s when the change propagates, about 200 ms when early cutoff stops it at the first layer (0.3.1; on a quiet host 0.3.0 was 68% slower than 0.2.0 there, 0.3.1 is 16% slower: see the 0.3.1 results).
Slice reads (one request each; the sum over a stored slice):
| Members | cold (first ever) | warm (memo) | fresh engine, validated | after one member's correction | memo per reader |
|---|---|---|---|---|---|
| 1,000 | 339 ms | 0.43 ms | 123 ms | 191 ms | 0.47 MB |
| 100,000 | 13.0 s | 0.58 ms | 7.9 s | 16.3 s | 46.7 MB |
A slice over a computed function of 1,000 members (each evaluated, signed and stored): 5.4 s cold, 0.54 ms warm, 453 ms validated by a fresh engine. (0.3.1 on store 0.3.2.) A stored slice read is one slice_values query (values, hashes and cells without receipt bytes) and, when the memo is on, one slice_changed_after query; 0.3.0 read the members' receipt bytes twice.
Licensed under Apache-2.0.