datumesh-core

The Rust core of the Datumesh protocol: strict JSON and RFC 8785 canonical bytes, cell URIs, key documents, response envelopes, unit resolution, and offline verification of signed receipts. It implements the specification (SPEC.md and units.md, revision spec-0.3.0) and passes every one of its 365 test vectors.

Example

[dependencies]
datumesh-core = "0.3.3"

Verify the receipt of spec/vectors/receipt/stored-number.json against the key documents the vector carries, then recompute it with the descriptor and value the vector's recompute object supplies:

use datumesh_core::json::{self, Value};
use datumesh_core::{verify, Cell, Keys, Level, MapResolver, Scalar};

fn main() {
    let vector = json::parse(&std::fs::read("stored-number.json").unwrap()).unwrap();
    let receipt = vector.get("input").and_then(Value::as_str).unwrap();

    // Keys are an input to verification, never fetched by it (SPEC 9.4). Here they come
    // from the vector; in an application, from your HTTP client's response passed to
    // `keys_from_response`, or from a cache.
    let mut keys = Keys::new();
    for (host, env) in vector.get("keys").and_then(Value::as_object).unwrap() {
        let body = env.get("https").and_then(|h| h.get("body")).and_then(Value::as_str);
        keys.insert_response(host, 200, body.map(str::as_bytes));
    }

    let result = verify(receipt.as_bytes(), &keys, None, None);
    assert!(result.ok, "{:?}", result.error());
    assert_eq!(result.level, Some(Level::Signed));
    println!("verified {} receipt {}", result.level.unwrap().as_str(), result.hash.unwrap());

    // Step 7 asks a Resolver for the descriptor of cell.fn_hash, the input receipts and the
    // evaluated value, only as far as it gets. MapResolver serves them from maps.
    let rc = vector.get("recompute").unwrap();
    let cell = Cell::from_value(json::parse(receipt.as_bytes()).unwrap().get("cell").unwrap()).unwrap();
    let mut resolver = MapResolver::new();
    resolver.insert_descriptor(cell.fn_hash.as_str(), rc.get("descriptor").unwrap().to_jcs());
    resolver.insert_value(cell.id(), Scalar::from_json(&rc.get("value").unwrap().to_jcs()).unwrap());
    let result = verify(receipt.as_bytes(), &keys, Some(&resolver), None);
    assert_eq!(result.level, Some(Level::Recomputed));
    println!("verified {} receipt {}", result.level.unwrap().as_str(), result.hash.unwrap());
}

It prints verified signed receipt 19a5fe3a744c5792f747fb4c286ce396ab44b95922de4f07937502fe595b6f81, then the same line with recomputed. A resolver that has the descriptor and inputs but cannot evaluate the function (evaluate returns None) leaves the receipt ok at level signed with code E_EVAL_UNAVAILABLE, the one code of SPEC 10.2 that is not a failure. Passing blob content runs step 7.7, after that too. Only None from descriptor means the descriptor is unavailable: offered text is parsed strictly, and null or any other value that is not the descriptor fails the hash check.

With feature sign, an engine makes receipts: SigningKey::from_seed (deterministic Ed25519, RFC 8032) and ReceiptBuilder, whose sign checks the members as Receipt::from_json would and derives computed_by from the signer host and the key. Signing the members of every vector receipt with the vector's seed reproduces its payload, signature and hash byte for byte.

What is in the crate

• json: strict parsing (SPEC 6.1) and RFC 8785 JCS, including ECMAScript number formatting: parse, Value, Value::to_jcs.
• Scalar, Cell, CellRef, Descriptor, Receipt, KeyDoc, Envelope, and canonicalize_*, cell_hash, function_hash, receipt_hash.
• parse_uri, format_uri, https_url: cell URIs and their HTTPS mapping.
• verify: SPEC 10.1 steps 1-7, with Ed25519 strictness as step 6 defines it; step 7 through a Resolver (MapResolver included).
• SigningKey and ReceiptBuilder behind the non-default sign feature.
• keys_from_response and Keys. fetch_keys (feature fetch) is deprecated since 0.3.1 and removed at 1.0: fetch https://<host>/.well-known/datumesh.json with your own HTTP client and pass the status and body to keys_from_response.
• testkit (non-default feature testkit): a conformance kit for Resolver implementations (see below).
• check_envelope: response envelopes, bound to the request: the cell id, and a receipt asof at or before the requested one (for ok and stale alike).
• resolved_unit: the unit of a cell's value.
• Error with an ErrorCode for every code of SPEC 10.2 (E_EVAL_UNAVAILABLE included), and a JSON path.

No unsafe, no panics on any input, no global state, no logging, no network outside the deprecated fetch_keys. Builds for wasm32-unknown-unknown, with or without sign and testkit. Minimum Rust version 1.97. The public API is frozen until 1.0 except for additions; API-CHANGES-0.3.0.md lists what changed from 0.2.1, API-CHANGES-0.3.1.md what was added in 0.3.1, API-CHANGES-0.3.2.md the (empty) diff of 0.3.2, API-CHANGES-0.3.3.md the additions in 0.3.3, and CHANGELOG.md the history.

Testing a resolver

A Resolver (a store, a cache, an engine's lookup) can check itself against the protocol with feature testkit, as a dev-dependency:

[dev-dependencies]
datumesh-core = { version = "0.3.3", features = ["testkit"] }

The kit never writes through the resolver. It gives you Fixtures (descriptors, receipts and the versions of stored cells, all canonical JSON, from the specification's vectors plus a few receipts signed for the kit), you load them, and it verifies receipts through your resolver and looks things up in it:

use datumesh_core::testkit::Kit;

#[test]
fn my_resolver_conforms() {
    let kit = Kit::new().time_aware(true); // false (the default) for one value per cell
    // A time-aware resolver is held to AnswerRule::TxAndAsof unless it says otherwise:
    // let kit = Kit::new().time_aware(true).answer_rule(datumesh_core::testkit::AnswerRule::TxOnly);
    if let Err(report) = kit.run(|fixtures| load_into_a_fresh_store(fixtures)) {
        panic!("{report}");
    }
}

It checks that verify asks the resolver only for what step 7 reaches (through the Recording wrapper), that what the resolver lacks is E_FN_UNAVAILABLE, E_INPUT_UNAVAILABLE or E_EVAL_UNAVAILABLE and that unknown or malformed hashes give None, that descriptor and receipt return exactly the bytes loaded, and that evaluate gets the receipt's asof; for a time-aware resolver, that a cell with versions at several times gives the latest version that answers as of the requested time under the kit's answer rule. Kit::answer_rule sets it: AnswerRule::TxAndAsof (the default) lets a version answer as of A only when its tx <= A and its receipt's asof <= A, as SPEC 5.5 step 5 (D31 revised) requires of an ok response to a request as of A; AnswerRule::TxOnly (the only rule before 0.3.3) only when its tx <= A. The fixtures' EUR/CHF version 1 (tx 2024-03-15T16:00, asof 2024-03-16T09:00) gives no value from its tx up to its asof under TxAndAsof. Each check (check_lazy_calls, check_unavailable, check_bytes, check_asof) can also run on its own. MapResolver passes it (not time-aware).

Repository

• crates/datumesh-core: the library.
• crates/datumesh-cli: datumesh-cli run-vectors (JSON lines in, one result per line out) and datumesh-cli check [DIR], for the cross-language harness. Not published.
• spec/: the specification, a git submodule.
• DEFECTS.md: every place the specification text was not enough to decide something.
• public-api.txt, public-api-all-features.txt: the public API (cargo public-api -ss) with default features and with --all-features; API-CHANGES-0.3.0.md: the diff against 0.2.1, justified; API-CHANGES-0.3.1.md: the additions in 0.3.1; API-CHANGES-0.3.2.md: the diff of 0.3.2 against 0.3.1; API-CHANGES-0.3.3.md: the additions in 0.3.3.
• crates/datumesh-core/src/testkit/data.rs: the kit's fixtures, generated from spec/ (and signed with feature sign) by tests/testkit_data.rs, which fails if they drift.
• fuzz/: cargo-fuzz targets. wasm-smoke/: the WebAssembly smoke test.
• tools/gen_tables.py: regenerates the embedded unit list and Unicode 15.1 table from spec/.

Licensed under Apache-2.0.