datumesh_http/lib.rs
1//! The [Datumesh](https://datumesh.com) engine behind HTTPS: the endpoints of a mesh
2//! (spec-0.4.1) over [`datumesh_engine::Engine`], and the `datumesh-mesh` server.
3//!
4//! A [`Mesh`] holds the engine, its store and what the endpoints share; [`Mesh::router`] is
5//! the axum router of every endpoint and [`Mesh::serve`] serves it. The blocking operations
6//! the endpoints are built on are library calls too ([`Mesh::cell`], [`Mesh::cells`],
7//! [`Mesh::resolve`], [`Mesh::verify`], [`call::resolve`], the cross-mesh resolver
8//! [`foreign::Foreign`]).
9//!
10//! # Endpoints
11//!
12//! | Endpoint | Answer |
13//! |---|---|
14//! | `GET /cell/<call or uri>` | the envelope (SPEC 5.5); `?asof=` for a call |
15//! | `POST /cells` | a JSON array of requests (calls or `cell://` URIs) to an array of envelopes, at most 1,000 |
16//! | `GET /slice/<call or uri with *>` | the members of a stored function's slice: JSON pages by key, or an Arrow IPC stream |
17//! | `GET /receipt/<hash>`, `/descriptor/<hash>`, `/manifest/<slice id>/<members hash>`, `/blob/<sha256>` | the bytes, immutable (also `/slice/<slice id>/<members hash>`, SPEC 15.13) |
18//! | `GET /explain/<call or uri>` | the lineage as nested envelopes, depth-bounded (`depth=`, default 4, at most 16) |
19//! | `GET /find?q=` | functions of the catalog by keyword (a stub until 2.2) |
20//! | `POST /verify` | the core verifier on a submitted receipt with this mesh's keys |
21//! | `GET /jobs/<id>` | a job: `pending`, or the envelope of the original request |
22//! | `GET /.well-known/datumesh.json`, `/.well-known/datumesh-mesh.json` | the key document (the store's namespace record for the host, SPEC 9.2) and the mesh document (SPEC 9.6; its URLs on [`Config::base_url`] when set: `http://` on a loopback address for an unrooted host only, local use, not valid under spec-0.4.3) |
23//! | `GET /bindings?name=&asof=` | which function a name denotes (SPEC 9.7), for names this mesh owns |
24//! | `GET /health`, `/metrics`, `/openapi.json` | liveness and the store; Prometheus text; the OpenAPI document generated from the handlers |
25//!
26//! `Accept` is honoured: JSON (and, at `/slice`, `application/vnd.apache.arrow.stream`); a
27//! request whose `Accept` excludes JSON is `406`, never an HTML page. Every response has a
28//! request id (`X-Request-Id`, taken from the request when it is a short token) and
29//! `Server-Timing`.
30//!
31//! # Requests and envelopes
32//!
33//! [`call`] says exactly which forms `/cell` accepts: a cell URI's rest byte for byte, or a
34//! read of SPEC 15.3 with literal arguments (positional by `signature.order`, or labelled),
35//! the callee a full name or an `@alias` of the configured map. A call is parsed by the
36//! formula crate's parser and written as its cell URI, so both are one request and give
37//! byte-identical envelopes; short names never appear in a response. Method chaining and
38//! key literals (SPEC 15.15, 1.3b) are `E_PARSE_URI` in 0.1.0.
39//!
40//! Every envelope is canonical JSON (JCS). Outcomes:
41//!
42//! | Outcome | When |
43//! |---|---|
44//! | `ok` | the engine's receipt; `trust` `primary` (signer's host is the namespace's, no inputs), `verified` (computed by this mesh), else `unverified` |
45//! | `stale` | the store or another mesh failed, and the mesh served a receipt for the cell before (`age_seconds` since) that answers the request |
46//! | `not_covered` | the name is not bound, the cell has no value; with `fault` when its evaluation faulted |
47//! | `unlicensed` | the configured policy names principals and the requester is not one; or the `Authorization` header is not a key of this mesh; or (since 0.2.1, spec-0.4.3) licence inheritance (SPEC 8.5) refuses the cell or a cell it reads (`EngineError::Unlicensed`) |
48//! | `payment_required` | the policy prices the function (a stub until 3.3) |
49//! | `rate_limited` | a rate limit refused the request (`retry_after_seconds`, `Retry-After`) |
50//! | `bad_request` | the request does not parse (every code of SPEC 5.4); `E_JOB_UNKNOWN` at a job URL |
51//! | `error` | the mesh failed: 503 for `E_INPUT_UNAVAILABLE`, `E_KEY_UNAVAILABLE`, `E_MESH_DOC`, `E_EVAL_UNAVAILABLE`; 502 with `detail` for `E_FOREIGN_UNAVAILABLE` (since 0.2.1: the key document of another mesh's function, which this mesh computes, gave no response); 500 otherwise |
52//! | `pending` | the answer took longer than the budget (1 s): poll `/jobs/<id>` |
53//!
54//! ## `Engine::get` as this crate uses it
55//!
56//! The mesh resolves the name itself (as of the request's asof, or now: the bindings it
57//! looked up are the cell's function hash) and calls `Engine::get_cell(cell, Some(A))`, A the
58//! request's asof or the clock reading at the request. The answer maps as the engine's table
59//! says: `Answer::Receipt` is `ok`; `Answer::NotCovered` `not_covered` (with `fault`);
60//! `EngineError::Invalid` with a code of SPEC 5.4 `bad_request`, with another code (a
61//! cross-mesh failure) `error` with that code; `NameUnresolved` `error` `E_FN_UNAVAILABLE`;
62//! `UndeclaredInput` `E_INPUT_UNDECLARED`; `Sign` its code (`E_TIME_ORDER`); `Store`
63//! `E_INPUT_UNAVAILABLE` (or `stale`); `Unlicensed` (since 0.2.1) `unlicensed`; every other failure (`NoRuntime`, `Runtime`, `Cycle`,
64//! `DepthExceeded`, `BudgetExceeded`, `WorkExceeded`) `E_EVAL_UNAVAILABLE`.
65//!
66//! # Licence inheritance and declarations (spec-0.4.3, since 0.2.1)
67//!
68//! The engine (0.4.1) signs a computed receipt under the licence SPEC 8.5 gives over its input
69//! receipts' licences, and refuses a cell the rule refuses: the mesh answers `unlicensed`
70//! (403, no new code, the function hash echoed; never cached as immutable, since a declaration
71//! may come later). The declarations it reads are those of the key document of the host of
72//! the cell's namespace, from the store. For this mesh's own host, that is the document it
73//! serves at `/.well-known/datumesh.json` (the store's record, unchanged: a mesh without
74//! declarations serves 0.2.0's bytes); `datumesh-mesh declare` (`keydoc::with_declaration`)
75//! adds or closes an entry, and `keydoc::with_key` keeps them through a rotation. Before it
76//! computes another mesh's function, the mesh puts that publisher's key document in the store
77//! (`foreign::Foreign::ensure_publisher`, the key document cache's TTL); a publisher that gives
78//! no HTTP response then is `error` `E_FOREIGN_UNAVAILABLE` (502, with `detail`).
79//!
80//! A mesh that serves `declarations` is unverifiable by clients on core < 0.3.7 or datumesh
81//! < 0.4.3; publishers should wait for their consumers to upgrade before declaring. The same
82//! holds for @datumesh/verify < 0.4.3.
83//!
84//! # Pending and jobs
85//!
86//! A request runs on the pool; if it has not finished within the budget, the mesh answers
87//! `pending` (202) with a job id of 256 bits from the operating system's CSPRNG,
88//! `estimate_seconds`, `poll` (`https://<host>/jobs/<id>`) and the original request, and the
89//! work goes on. `GET /jobs/<id>` answers `pending` again while it runs, then the envelope the
90//! `/cell` request would have had, with its status, bound to the original request. Jobs and
91//! their results are kept while the mesh runs (SPEC 5.5, spec-0.4.2): 300 s after a result is
92//! first served (and 300 s after completion if never served); an
93//! unknown or expired id is `bad_request` `E_JOB_UNKNOWN`. Jobs are held in memory: a
94//! restart forgets them, and the client's remedy for `E_JOB_UNKNOWN` (send the original
95//! request again) is then answered from the store, where the evaluation's receipt was kept.
96//! A batch's requests past the budget become jobs one by one.
97//!
98//! # Cross-mesh resolution
99//!
100//! See [`foreign`]: names under a host this mesh does not own are imported into the store
101//! (bindings, descriptors, the cells' receipts, verified against the signer's key document),
102//! through a layer around every runtime, so the engine reads foreign cells as it reads its
103//! own. Lifetimes: key documents 300 s (a key id not in the cached document refetches it at
104//! once), mesh documents 300 s, a foreign name's bindings and a foreign cell's latest
105//! version confirmed again after 60 s; descriptors and receipts forever (content-addressed);
106//! answers as of an instant before the last confirmation never change. The other mesh down:
107//! `error` (`E_INPUT_UNAVAILABLE`), or `stale` for a cell answered before; never
108//! `not_covered`.
109//!
110//! # Caching
111//!
112//! `ok` as of a past instant, and receipts, descriptors, manifests and blobs:
113//! `Cache-Control: public, max-age=31536000, immutable`. Answers about now: `public,
114//! max-age=10` and `Link: <https://<host>/cell/<uri with asof>>; rel="canonical"`, the as-of
115//! form of the same answer. Answers that depend on who asks (a policy applies): `private,
116//! no-store`. Errors, refusals and jobs: `no-store`. Responses are gzipped when the client
117//! asks (`Accept-Encoding: gzip`).
118//!
119//! # Concurrency and rate limits
120//!
121//! axum on tokio: the engine and the store are synchronous, so every request's work runs on
122//! a `spawn_blocking` thread holding one of [`Config::pool_size`] permits (default 32). The
123//! pool's size is the concurrency limit: it is the store's connection pool size too (the
124//! server connects with that many connections), large enough that store latency (about
125//! 0.2 ms a query) does not idle the 6 cores of the target VM and small enough that
126//! PostgreSQL's default 100 connections are never approached; it is reported in `/metrics`
127//! (`datumesh_pool_size`, `_in_use`, `_peak`, `_waited_total`). Rate limits are token buckets
128//! checked before any pool work, per IP (60 a minute, burst 120), per API key (600 a minute)
129//! and for `POST /verify` per IP (60 a minute); a refused request is answered at once, never
130//! queued.
131//!
132//! # Secrets and logs
133//!
134//! The signing key and the API keys file are read only from files with mode 0600
135//! ([`config::read_secret_file`]); API keys are held as SHA-256 hashes. Logs (`tracing`,
136//! JSON by default in the server) carry a request id, the route template, the status, the
137//! latency, the client IP and the principal id; never a key, a job id or a path.
138
139#![forbid(unsafe_code)]
140#![deny(missing_docs)]
141
142pub mod apikeys;
143pub mod call;
144pub mod client;
145pub mod config;
146mod describe;
147pub mod domain;
148pub mod envelope;
149pub mod foreign;
150mod http;
151mod jobs;
152pub mod keydoc;
153mod limit;
154mod mesh;
155mod metrics;
156mod pool;
157pub mod publish;
158pub mod slice;
159pub mod vectors;
160
161pub use config::Config;
162pub use http::openapi;
163pub use mesh::{BuildError, Mesh, MeshBuilder, Verified};