datumesh_http/lib.rs
1//! The [Datumesh](https://datumesh.com) engine behind HTTPS: the endpoints of a mesh
2//! (spec-0.4.1) over [`datumesh_engine::Engine`], and the `datumesh-mesh` server.
3//!
4//! A [`Mesh`] holds the engine, its store and what the endpoints share; [`Mesh::router`] is
5//! the axum router of every endpoint and [`Mesh::serve`] serves it. The blocking operations
6//! the endpoints are built on are library calls too ([`Mesh::cell`], [`Mesh::cells`],
7//! [`Mesh::resolve`], [`Mesh::verify`], [`call::resolve`], the cross-mesh resolver
8//! [`foreign::Foreign`]).
9//!
10//! # Endpoints
11//!
12//! | Endpoint | Answer |
13//! |---|---|
14//! | `GET /cell/<call or uri>` | the envelope (SPEC 5.5); `?asof=` for a call |
15//! | `POST /cells` | a JSON array of requests (calls or `cell://` URIs) to an array of envelopes, at most 1,000 |
16//! | `GET /slice/<call or uri with *>` | the members of a stored function's slice: JSON pages by key, or an Arrow IPC stream |
17//! | `GET /receipt/<hash>`, `/descriptor/<hash>`, `/manifest/<slice id>/<members hash>`, `/blob/<sha256>` | the bytes, immutable (also `/slice/<slice id>/<members hash>`, SPEC 15.13) |
18//! | `GET /explain/<call or uri>` | the lineage as nested envelopes, depth-bounded (`depth=`, default 4, at most 16) |
19//! | `GET /find?q=` | functions of the catalog by keyword (a stub until 2.2) |
20//! | `POST /verify` | the core verifier on a submitted receipt with this mesh's keys |
21//! | `GET /jobs/<id>` | a job: `pending`, or the envelope of the original request |
22//! | `GET /.well-known/datumesh.json`, `/.well-known/datumesh-mesh.json` | the key document (the store's namespace record for the host, SPEC 9.2) and the mesh document (SPEC 9.6; its URLs on [`Config::base_url`] when set: `http://` on a loopback address for an unrooted host only, local use, not valid under spec-0.4.3) |
23//! | `GET /bindings?name=&asof=` | which function a name denotes (SPEC 9.7), for names this mesh owns |
24//! | `GET /health`, `/metrics`, `/openapi.json` | liveness and the store; Prometheus text; the OpenAPI document generated from the handlers |
25//!
26//! `Accept` is honoured: JSON (and, at `/slice`, `application/vnd.apache.arrow.stream`); a
27//! request whose `Accept` excludes JSON is `406`, never an HTML page. Every response has a
28//! request id (`X-Request-Id`, taken from the request when it is a short token) and
29//! `Server-Timing`.
30//!
31//! # Requests and envelopes
32//!
33//! [`call`] says exactly which forms `/cell` accepts: a cell URI's rest byte for byte, or a
34//! read of SPEC 15.3 with literal arguments (positional by `signature.order`, or labelled),
35//! the callee a full name or an `@alias` of the configured map. A call is parsed by the
36//! formula crate's parser and written as its cell URI, so both are one request and give
37//! byte-identical envelopes; short names never appear in a response. Method chaining and
38//! key literals (SPEC 15.15, 1.3b) are `E_PARSE_URI` in 0.1.0.
39//!
40//! Every envelope is canonical JSON (JCS). Outcomes:
41//!
42//! | Outcome | When |
43//! |---|---|
44//! | `ok` | the engine's receipt; `trust` `primary` (signer's host is the namespace's, no inputs), `verified` (computed by this mesh), else `unverified` |
45//! | `stale` | the store or another mesh failed, and the mesh served a receipt for the cell before (`age_seconds` since) that answers the request |
46//! | `not_covered` | the name is not bound, the cell has no value; with `fault` when its evaluation faulted |
47//! | `unlicensed` | the configured policy names principals and the requester is not one; or the `Authorization` header is not a key of this mesh |
48//! | `payment_required` | the policy prices the function (a stub until 3.3) |
49//! | `rate_limited` | a rate limit refused the request (`retry_after_seconds`, `Retry-After`) |
50//! | `bad_request` | the request does not parse (every code of SPEC 5.4); `E_JOB_UNKNOWN` at a job URL |
51//! | `error` | the mesh failed: 503 for `E_INPUT_UNAVAILABLE`, `E_KEY_UNAVAILABLE`, `E_MESH_DOC`, `E_EVAL_UNAVAILABLE`; 500 otherwise |
52//! | `pending` | the answer took longer than the budget (1 s): poll `/jobs/<id>` |
53//!
54//! ## `Engine::get` as this crate uses it
55//!
56//! The mesh resolves the name itself (as of the request's asof, or now: the bindings it
57//! looked up are the cell's function hash) and calls `Engine::get_cell(cell, Some(A))`, A the
58//! request's asof or the clock reading at the request. The answer maps as the engine's table
59//! says: `Answer::Receipt` is `ok`; `Answer::NotCovered` `not_covered` (with `fault`);
60//! `EngineError::Invalid` with a code of SPEC 5.4 `bad_request`, with another code (a
61//! cross-mesh failure) `error` with that code; `NameUnresolved` `error` `E_FN_UNAVAILABLE`;
62//! `UndeclaredInput` `E_INPUT_UNDECLARED`; `Sign` its code (`E_TIME_ORDER`); `Store`
63//! `E_INPUT_UNAVAILABLE` (or `stale`); every other failure (`NoRuntime`, `Runtime`, `Cycle`,
64//! `DepthExceeded`, `BudgetExceeded`, `WorkExceeded`) `E_EVAL_UNAVAILABLE`.
65//!
66//! # Pending and jobs
67//!
68//! A request runs on the pool; if it has not finished within the budget, the mesh answers
69//! `pending` (202) with a job id of 256 bits from the operating system's CSPRNG,
70//! `estimate_seconds`, `poll` (`https://<host>/jobs/<id>`) and the original request, and the
71//! work goes on. `GET /jobs/<id>` answers `pending` again while it runs, then the envelope the
72//! `/cell` request would have had, with its status, bound to the original request. Jobs and
73//! their results are kept while the mesh runs (SPEC 5.5, spec-0.4.2): 300 s after a result is
74//! first served (and 300 s after completion if never served); an
75//! unknown or expired id is `bad_request` `E_JOB_UNKNOWN`. Jobs are held in memory: a
76//! restart forgets them, and the client's remedy for `E_JOB_UNKNOWN` (send the original
77//! request again) is then answered from the store, where the evaluation's receipt was kept.
78//! A batch's requests past the budget become jobs one by one.
79//!
80//! # Cross-mesh resolution
81//!
82//! See [`foreign`]: names under a host this mesh does not own are imported into the store
83//! (bindings, descriptors, the cells' receipts, verified against the signer's key document),
84//! through a layer around every runtime, so the engine reads foreign cells as it reads its
85//! own. Lifetimes: key documents 300 s (a key id not in the cached document refetches it at
86//! once), mesh documents 300 s, a foreign name's bindings and a foreign cell's latest
87//! version confirmed again after 60 s; descriptors and receipts forever (content-addressed);
88//! answers as of an instant before the last confirmation never change. The other mesh down:
89//! `error` (`E_INPUT_UNAVAILABLE`), or `stale` for a cell answered before; never
90//! `not_covered`.
91//!
92//! # Caching
93//!
94//! `ok` as of a past instant, and receipts, descriptors, manifests and blobs:
95//! `Cache-Control: public, max-age=31536000, immutable`. Answers about now: `public,
96//! max-age=10` and `Link: <https://<host>/cell/<uri with asof>>; rel="canonical"`, the as-of
97//! form of the same answer. Answers that depend on who asks (a policy applies): `private,
98//! no-store`. Errors, refusals and jobs: `no-store`. Responses are gzipped when the client
99//! asks (`Accept-Encoding: gzip`).
100//!
101//! # Concurrency and rate limits
102//!
103//! axum on tokio: the engine and the store are synchronous, so every request's work runs on
104//! a `spawn_blocking` thread holding one of [`Config::pool_size`] permits (default 32). The
105//! pool's size is the concurrency limit: it is the store's connection pool size too (the
106//! server connects with that many connections), large enough that store latency (about
107//! 0.2 ms a query) does not idle the 6 cores of the target VM and small enough that
108//! PostgreSQL's default 100 connections are never approached; it is reported in `/metrics`
109//! (`datumesh_pool_size`, `_in_use`, `_peak`, `_waited_total`). Rate limits are token buckets
110//! checked before any pool work, per IP (60 a minute, burst 120), per API key (600 a minute)
111//! and for `POST /verify` per IP (60 a minute); a refused request is answered at once, never
112//! queued.
113//!
114//! # Secrets and logs
115//!
116//! The signing key and the API keys file are read only from files with mode 0600
117//! ([`config::read_secret_file`]); API keys are held as SHA-256 hashes. Logs (`tracing`,
118//! JSON by default in the server) carry a request id, the route template, the status, the
119//! latency, the client IP and the principal id; never a key, a job id or a path.
120
121#![forbid(unsafe_code)]
122#![deny(missing_docs)]
123
124pub mod apikeys;
125pub mod call;
126pub mod client;
127pub mod config;
128pub mod envelope;
129pub mod foreign;
130mod http;
131mod jobs;
132pub mod keydoc;
133mod limit;
134mod mesh;
135mod metrics;
136mod pool;
137pub mod publish;
138pub mod slice;
139pub mod vectors;
140
141pub use config::Config;
142pub use http::openapi;
143pub use mesh::{BuildError, Mesh, MeshBuilder, Verified};