Skip to main content

Crate datumesh_http

Crate datumesh_http 

Source
Expand description

The Datumesh engine behind HTTPS: the endpoints of a mesh (spec-0.4.1) over datumesh_engine::Engine, and the datumesh-mesh server.

A Mesh holds the engine, its store and what the endpoints share; Mesh::router is the axum router of every endpoint and Mesh::serve serves it. The blocking operations the endpoints are built on are library calls too (Mesh::cell, Mesh::cells, Mesh::resolve, Mesh::verify, call::resolve, the cross-mesh resolver foreign::Foreign).

§Endpoints

EndpointAnswer
GET /cell/<call or uri>the envelope (SPEC 5.5); ?asof= for a call
POST /cellsa JSON array of requests (calls or cell:// URIs) to an array of envelopes, at most 1,000
GET /slice/<call or uri with *>the members of a stored function’s slice: JSON pages by key, or an Arrow IPC stream
GET /receipt/<hash>, /descriptor/<hash>, /manifest/<slice id>/<members hash>, /blob/<sha256>the bytes, immutable (also /slice/<slice id>/<members hash>, SPEC 15.13)
GET /explain/<call or uri>the lineage as nested envelopes, depth-bounded (depth=, default 4, at most 16)
GET /find?q=functions of the catalog by keyword (a stub until 2.2)
POST /verifythe core verifier on a submitted receipt with this mesh’s keys
GET /jobs/<id>a job: pending, or the envelope of the original request
GET /.well-known/datumesh.json, /.well-known/datumesh-mesh.jsonthe key document (the store’s namespace record for the host, SPEC 9.2) and the mesh document (SPEC 9.6; its URLs on Config::base_url when set: http:// on a loopback address for an unrooted host only, local use, not valid under spec-0.4.3)
GET /bindings?name=&asof=which function a name denotes (SPEC 9.7), for names this mesh owns
GET /health, /metrics, /openapi.jsonliveness and the store; Prometheus text; the OpenAPI document generated from the handlers

Accept is honoured: JSON (and, at /slice, application/vnd.apache.arrow.stream); a request whose Accept excludes JSON is 406, never an HTML page. Every response has a request id (X-Request-Id, taken from the request when it is a short token) and Server-Timing.

§Requests and envelopes

call says exactly which forms /cell accepts: a cell URI’s rest byte for byte, or a read of SPEC 15.3 with literal arguments (positional by signature.order, or labelled), the callee a full name or an @alias of the configured map. A call is parsed by the formula crate’s parser and written as its cell URI, so both are one request and give byte-identical envelopes; short names never appear in a response. Method chaining and key literals (SPEC 15.15, 1.3b) are E_PARSE_URI in 0.1.0.

Every envelope is canonical JSON (JCS). Outcomes:

OutcomeWhen
okthe engine’s receipt; trust primary (signer’s host is the namespace’s, no inputs), verified (computed by this mesh), else unverified
stalethe store or another mesh failed, and the mesh served a receipt for the cell before (age_seconds since) that answers the request
not_coveredthe name is not bound, the cell has no value; with fault when its evaluation faulted
unlicensedthe configured policy names principals and the requester is not one; or the Authorization header is not a key of this mesh; or (since 0.2.1, spec-0.4.3) licence inheritance (SPEC 8.5) refuses the cell or a cell it reads (EngineError::Unlicensed)
payment_requiredthe policy prices the function (a stub until 3.3)
rate_limiteda rate limit refused the request (retry_after_seconds, Retry-After)
bad_requestthe request does not parse (every code of SPEC 5.4); E_JOB_UNKNOWN at a job URL
errorthe mesh failed: 503 for E_INPUT_UNAVAILABLE, E_KEY_UNAVAILABLE, E_MESH_DOC, E_EVAL_UNAVAILABLE; 502 with detail for E_FOREIGN_UNAVAILABLE (since 0.2.1: the key document of another mesh’s function, which this mesh computes, gave no response); 500 otherwise
pendingthe answer took longer than the budget (1 s): poll /jobs/<id>

§Engine::get as this crate uses it

The mesh resolves the name itself (as of the request’s asof, or now: the bindings it looked up are the cell’s function hash) and calls Engine::get_cell(cell, Some(A)), A the request’s asof or the clock reading at the request. The answer maps as the engine’s table says: Answer::Receipt is ok; Answer::NotCovered not_covered (with fault); EngineError::Invalid with a code of SPEC 5.4 bad_request, with another code (a cross-mesh failure) error with that code; NameUnresolved error E_FN_UNAVAILABLE; UndeclaredInput E_INPUT_UNDECLARED; Sign its code (E_TIME_ORDER); Store E_INPUT_UNAVAILABLE (or stale); Unlicensed (since 0.2.1) unlicensed; every other failure (NoRuntime, Runtime, Cycle, DepthExceeded, BudgetExceeded, WorkExceeded) E_EVAL_UNAVAILABLE.

§Licence inheritance and declarations (spec-0.4.3, since 0.2.1)

The engine (0.4.1) signs a computed receipt under the licence SPEC 8.5 gives over its input receipts’ licences, and refuses a cell the rule refuses: the mesh answers unlicensed (403, no new code, the function hash echoed; never cached as immutable, since a declaration may come later). The declarations it reads are those of the key document of the host of the cell’s namespace, from the store. For this mesh’s own host, that is the document it serves at /.well-known/datumesh.json (the store’s record, unchanged: a mesh without declarations serves 0.2.0’s bytes); datumesh-mesh declare (keydoc::with_declaration) adds or closes an entry, and keydoc::with_key keeps them through a rotation. Before it computes another mesh’s function, the mesh puts that publisher’s key document in the store (foreign::Foreign::ensure_publisher, the key document cache’s TTL); a publisher that gives no HTTP response then is error E_FOREIGN_UNAVAILABLE (502, with detail).

A mesh that serves declarations is unverifiable by clients on core < 0.3.7 or datumesh < 0.4.3; publishers should wait for their consumers to upgrade before declaring. The same holds for @datumesh/verify < 0.4.3.

§Pending and jobs

A request runs on the pool; if it has not finished within the budget, the mesh answers pending (202) with a job id of 256 bits from the operating system’s CSPRNG, estimate_seconds, poll (https://<host>/jobs/<id>) and the original request, and the work goes on. GET /jobs/<id> answers pending again while it runs, then the envelope the /cell request would have had, with its status, bound to the original request. Jobs and their results are kept while the mesh runs (SPEC 5.5, spec-0.4.2): 300 s after a result is first served (and 300 s after completion if never served); an unknown or expired id is bad_request E_JOB_UNKNOWN. Jobs are held in memory: a restart forgets them, and the client’s remedy for E_JOB_UNKNOWN (send the original request again) is then answered from the store, where the evaluation’s receipt was kept. A batch’s requests past the budget become jobs one by one.

§Cross-mesh resolution

See foreign: names under a host this mesh does not own are imported into the store (bindings, descriptors, the cells’ receipts, verified against the signer’s key document), through a layer around every runtime, so the engine reads foreign cells as it reads its own. Lifetimes: key documents 300 s (a key id not in the cached document refetches it at once), mesh documents 300 s, a foreign name’s bindings and a foreign cell’s latest version confirmed again after 60 s; descriptors and receipts forever (content-addressed); answers as of an instant before the last confirmation never change. The other mesh down: error (E_INPUT_UNAVAILABLE), or stale for a cell answered before; never not_covered.

§Caching

ok as of a past instant, and receipts, descriptors, manifests and blobs: Cache-Control: public, max-age=31536000, immutable. Answers about now: public, max-age=10 and Link: <https://<host>/cell/<uri with asof>>; rel="canonical", the as-of form of the same answer. Answers that depend on who asks (a policy applies): private, no-store. Errors, refusals and jobs: no-store. Responses are gzipped when the client asks (Accept-Encoding: gzip).

§Concurrency and rate limits

axum on tokio: the engine and the store are synchronous, so every request’s work runs on a spawn_blocking thread holding one of Config::pool_size permits (default 32). The pool’s size is the concurrency limit: it is the store’s connection pool size too (the server connects with that many connections), large enough that store latency (about 0.2 ms a query) does not idle the 6 cores of the target VM and small enough that PostgreSQL’s default 100 connections are never approached; it is reported in /metrics (datumesh_pool_size, _in_use, _peak, _waited_total). Rate limits are token buckets checked before any pool work, per IP (60 a minute, burst 120), per API key (600 a minute) and for POST /verify per IP (60 a minute); a refused request is answered at once, never queued.

§Secrets and logs

The signing key and the API keys file are read only from files with mode 0600 (config::read_secret_file); API keys are held as SHA-256 hashes. Logs (tracing, JSON by default in the server) carry a request id, the route template, the status, the latency, the client IP and the principal id; never a key, a job id or a path.

Re-exports§

pub use config::Config;

Modules§

apikeys
API keys: Authorization: Bearer dm_<random>.
call
What a request names: a cell URI’s <rest> (SPEC 5.3) or a call (SPEC 5.8), resolved to one canonical reference, pinned to a function hash or not (spec-0.4.2).
client
The store-less client of other meshes (for datumesh-local and for this mesh’s cross-mesh reads): fetch a host’s key document and mesh document, its bindings, a cell’s envelope (by call, URI or pin), slice pages, batches, receipts, descriptors, manifests, blobs and versions, and check each with the core: the envelope against the request (check_answer, check_slice_answer, check_batch), every receipt’s signature against its signer’s key document (fetched from the signer’s own host, never from the mesh that served it), and content-addressed bytes against their hash. Nothing is stored; the import into a store is crate::foreign::Foreign’s, a layer on top.
config
The mesh’s configuration: Config, from code or from the environment (Config::from_env).
domain
Domain validation (since the namespace-validation step): a namespace host is valid when https://<host>/.well-known/datumesh.json fetches and passes the key-document rules (SPEC 9.2, 9.3: datumesh_core::key_document_from_response, declarations included).
envelope
Envelopes (SPEC 5.5) as the mesh writes them, and Reply: an HTTP answer before it is an HTTP response.
foreign
Cross-mesh resolution (SPEC 9.6, 9.7), minimal form: names under hosts this mesh does not own.
keydoc
Key documents (SPEC 9.2) for the mesh’s own host, rotation (SPEC 9.5), and since 0.2.1 their declarations (spec-0.4.3).
publish
Publishing stored cells as the mesh (datumesh-mesh publish-stored): a JSON lines file of stored functions and their cells, signed with the mesh’s key, put in the store.
slice
GET /slice/<call with *>: the members of a stored function’s slice as of an instant, as an Arrow IPC stream or as JSON pages, paginated by key.
vectors
import-vectors: the spec’s vectors’ cells into a store (see deploy/README.md).

Structs§

Mesh
A Datumesh mesh: the engine behind the endpoints. Cheap to clone (shared).
MeshBuilder
Builds a Mesh.
Verified
What POST /verify reports.

Enums§

BuildError
Why a mesh could not be built.

Functions§

openapi
The OpenAPI document (generated from the handlers’ annotations).