Crate mkit_server

Expand description

§mkit-server

A generic, runtime-agnostic mkit server core. Embedders build a deployment on it (for example, the Cloudflare Workers adapter in this repository), and mkit serve in the mkit CLI uses its fs and ssh pieces. It compiles for native targets and wasm32-unknown-unknown.

§What it provides

Identifiers (RepoId, NamespaceKey, Addressing), principals and the typed Operation model.
A transport-neutral ServerError with redaction by construction, mapped to Connect and ssh error codes by the bindings.
The runtime model: MaybeSend, BoxFuture, injected Clock and Spawner, and send_wrap; plus a Metrics facade with no metrics backend.
The storage contract (store): a key-level NamespaceStore whose only write is one declarative Batch, a content-addressed BlobStore, key layouts, value codecs and the replay-ledger model.
The request pipeline (pipeline): stage traits, auth modes (open, bearer, auth v2, transport identity), shard routing, the unary RPCs, and resumable streaming upload and download.
Protocol helpers shared by every binding: ref CAS and name checks, upload and download framing, quota evaluation and storage-error redaction.

§Features

Feature	Effect
`connect` (default)	The `mkit.transport.v1` Connect binding (`connect::service`), with its health service and auth interceptor. wasm-clean.
`memory`	In-memory reference backends, a template for third-party backends.
`fs`	std-only stores over the `.mkit` on-disk layout (`FsBlobStore`, `FsLayoutStore`). Native only.
`sql`	A shared SQL backend (`SqlKvStore`) over a synchronous `SqlConn` trait; the embedder supplies the engine.
`ssh`	The `mkit.rpc.v1.ssh` session (`ssh::serve_session`), with no async runtime of its own.
`remote-hooks`	Signed `mkit.server.hooks.v1` authorization, admission and outcome adapters over a `HookChannel`.
`http-objects`	Runtime-agnostic HTTP object serving; requires explicit configuration and indexed mode.
`published-view`	Published reader source; explicit adapter configuration.
`pack-ruzstd`	Pure-Rust zstd decoding for wasm targets.
`test-faults`	Test-only fault injection. Never enable it in a release build.

mkit-server-worker: Cloudflare Workers adapter (R2 blobs, Durable Objects). Not published.
mkit-server-conformance: backend and wire conformance suites. Not published.

See the crate documentation and docs/SPEC-SERVER.md in the repository for details.

§License

Licensed under either of Apache License, Version 2.0 or MIT license, at your option.

This crate is runtime-agnostic: it compiles for the host and for wasm32-unknown-unknown, never reads the wall clock directly (see Clock) and never spawns on a concrete executor (see Spawner). The shared vocabulary is re-exported at the crate root from private modules. The protocol logic lives in public modules, so call sites name the protocol they apply (refs::evaluate_cas, quota::evaluate_quota). The storage contract lives in store: its contract types are also re-exported at the root, its key layouts, value codecs and typed readers stay namespaced (store::keys, store::codec, store::read), as do the export/import helpers and the ContentIndex row types (store::export_partition, store::Holder). The memory feature adds the in-memory reference backends; the native fs feature adds the fs module, the stores over the .mkit on-disk layout; the sql feature adds sql::SqlKvStore, the store over any synchronous sql::SqlConn; the ssh feature adds the ssh module, the mkit.rpc.v1.ssh session over the pipeline. The connect feature (default) adds the mkit.transport.v1 Connect binding over the pipeline (connect::service); the remote-hooks feature adds the hooks module, the mkit.server.hooks.v1 adapter over a transport-agnostic channel; the http-objects feature adds the http_objects module and Pipeline::serve_http_object (explicit indexed HTTP opt-in).

Re-exports§

pub use policy::GrantConfig;
pub use store::Batch;
pub use store::BatchOutcome;
pub use store::BlobBody;
pub use store::BlobKey;
pub use store::BlobMeta;
pub use store::BlobNamespace;
pub use store::BlobStore;
pub use store::BoxError;
pub use store::ByteRange;
pub use store::CommitOutcome;
pub use store::ContentIndex;
pub use store::Cursor;
pub use store::Key;
pub use store::KeyClasses;
pub use store::MAX_BATCH_BYTES;
pub use store::MAX_BATCH_OPS;
pub use store::MAX_KEY_BYTES;
pub use store::MAX_SCAN_RANGES;
pub use store::MAX_VALUE_BYTES;
pub use store::MembershipMode;
pub use store::MultipartBlobStore;
pub use store::NamespaceStore;
pub use store::PackSink;
pub use store::PartRef;
pub use store::PartSink;
pub use store::Partition;
pub use store::PartitionStats;
pub use store::Precondition;
pub use store::RangeScan;
pub use store::ScanPage;
pub use store::StateCommitment;
pub use store::StoreCapabilities;
pub use store::StoreError;
pub use store::StoreMaintenance;
pub use store::UnsupportedPartSink;
pub use store::Value;
pub use store::Write;
pub use store::is_reserved_pack_keyspace;
pub use telemetry::METRIC_LATENCY;
pub use telemetry::METRIC_REQUESTS;
pub use telemetry::METRIC_UPLOAD_BYTES;
pub use telemetry::Metrics;
pub use telemetry::NEVER_ECHO;
pub use telemetry::NEVER_LOG;
pub use telemetry::NoopMetrics;
pub use telemetry::REDACTED_VALUE;
pub use telemetry::Redactor;
pub use telemetry::is_never_echo;
pub use telemetry::is_never_log;

Modules§

admin: Signed operator API (§16), durable replay and a gapless audit chain.
auth_v2: Auth v2 glue (SPEC-TRANSPORT-CONNECT §7.1): read the ten headers, run mkit_core::write_auth::verify_headers and decode its result into a VerifiedAuth. Verification itself (canonical fields, validity window, strict Ed25519) stays in mkit-core; nothing here reimplements it.
authority: Deployment-authority generation statements (SPEC-SERVER §6.2.1). The outer RPC is unsigned; this bounded statement is its authorization.
connect: The mkit.transport.v1 Connect binding (feature connect, SPEC-TRANSPORT-CONNECT): service mounts TransportService and grpc.health.v1.Health over a Pipeline, behind the AuthInterceptor. It is runtime-agnostic and wasm-clean (connectrpc without its server and zstd features), so the native adapter serves it through axum and the Workers adapter through its fetch bridge, both unchanged.
download: DownloadPack chunking (SPEC-TRANSPORT-CONNECT §6.2).
fs: Stores over the .mkit on-disk layout that FileTransport uses today (PRD §5.1, §6.9): FsBlobStore for <root>/packs/<64-hex> and FsLayoutStore for refs as files under <root>/refs/, so mkit serve <repo-path> (ssh) and any mkit-server fs-layout deployment serve the same files local mkit commands and mkit+file:// remotes read.
hooks: Remote hooks: the mkit.server.hooks.v1 adapter (SPEC-SERVER §§6-8), behind the remote-hooks feature.
http_objects: Runtime-agnostic HTTP object serving (SPEC-HTTP-OBJECTS; WP-4.12, R-169).
indexed: Indexed ingestion shared by native inline verification and future Worker scheduling. Verification extracts large objects into the global object store before a pack’s Verified state is written (WP-4.10).
pipeline: The transport-neutral request pipeline (PRD §5.4): the unary RPCs of mkit.transport.v1 over the storage contract.
policy: Deployment namespace and write policy (SPEC-TRANSPORT-CONNECT §7.5).
purge: Durable cache purge work and deployment-independent selectors (§16.7).
quota: Write quotas: the value types, the default limits and the fixed-window evaluation.
refs: Ref compare-and-swap and ref-name helpers (SPEC-TRANSPORT-CONNECT §3, SPEC-TRANSPORT §4.2.1, SPEC-REFS §3 and §4).
relay: Source-side, at-least-once outbox delivery. Target watermarks make redelivery idempotent, including a crash before source cleanup.
scanner_retrieval: Private raw-pack retrieval for synchronous scanners (SPEC-SERVER §11.4). Capabilities authorize only open ticket-bound bytes; public serving is unrelated.
sql: The shared SQL backend (sql feature): the NamespaceStore contract implemented once over SQLite, on the tiny synchronous SqlConn trait.
ssh: The mkit.rpc.v1.ssh session over the pipeline (PRD §6.9, D23; SPEC-TRANSPORT §4.2): the Hello handshake, per-verb dispatch, the streaming upload and download, the per-connection budgets and the CAS conflict reply of mkit serve, moved here from mkit-cli so the ssh stdio path and any enc listener share one implementation.
storage_error: Storage-failure redaction (issue #794).
store: The storage contract (PRD §5.3): the key-level NamespaceStore, the content-addressed BlobStore, and the key layouts, value codecs and typed readers every backend shares. Nothing here needs SQL: a single-writer key-value store implements the whole contract.
takedown: Lean denial and durable takedown intents; adapters control activation.
telemetry: Metrics facade and logging hygiene (PRD §8 M0: tracing, metrics, redaction).
timers: Partition-local timers, shared by native and Durable Object drivers.
upload: UploadPack stream framing (SPEC-TRANSPORT-CONNECT §6.1, SPEC-TRANSPORT §4.2).
url_token: Signed URL tokens (SPEC-WRITE-GRANTS §9.4): the mkit-url-token:v1 statement, its <statement>.<signature> encoding, the deployment’s Ed25519 key set and IssueObjectUrl’s mint.

Structs§

AuthzFacts: Facts the Authorizer established, carried into apply as preconditions. M1 establishes owner; M2 (WP-2.6) sets grant so the pipeline can require grant.epoch when it commits.
Creation: Namespace and repository creation facts for a write.
Elapsed: with_timeout gave up before the future finished.
ErrorDetail: A typed error detail (Connect ErrorDetail): the fully qualified proto message name and its encoded bytes.
GrantRef: A grant the Authorizer matched, with the namespace epoch it was checked against.
ManualClock: A clock that only moves when told to. Usable on every target, and shared across threads through interior mutability.
ManualSleep: A Sleep that only completes when told to, for tests. Shared across clones; ManualSleep::elapsed starts already fired.
MemoryBlobStore: An in-memory BlobStore for one keyspace. Clones share the blobs.
MemoryKv: An in-memory NamespaceStore over a BTreeMap per partition.
MemoryPackSink: The upload handle of MemoryBlobStore. It hashes incrementally and buffers the blob: the one exception to PackSink’s one-part memory bound, since this backend holds every blob in memory anyway.
MultiAddressing: Multi-repository addressing with a deployment namespace policy.
NamespaceKey: A namespace key. Under D34 it selects the namespace coordinator and prefixes every shard key. Single deployments use the reserved default; Multi deployments use a parsed self-certifying namespace (§7.4).
Operation: A decoded request, ready for policy and storage.
Redacted: A string that never prints its contents through fmt::Debug or fmt::Display. Only Redacted::expose reads it, and only the server-side logging sink should call that.
RefUpdate: One conditional ref write.
ReplayKey: The replay key: auth v2 Authorized.scope, BLAKE3(audience \n repository \n pubkey \n nonce).
ReplayRecord: A replay-ledger record.
RepoId: A repository’s full identity.
RepoName: A repository name: 1..=255 bytes of printable ASCII with no whitespace (bytes 0x21..=0x7e).
ResolvedRepo: A resolved request target and its byte-exact wire identity.
ServerError: A transport-neutral server error. Display shows the public message only. Debug shows the log detail as Redacted and replaces the value of every crate::NEVER_LOG header with a placeholder; a sink that knows deployment-configured names logs headers through crate::Redactor instead.
StoredRejection: A storable rejection: a final code and its public message, with no error detail (so never an admission challenge).
SystemClock: The host wall clock. Native targets only; Workers supply their own.
VerifiedAuth: A verified auth v2 authorization, decoded from mkit_core::write_auth::Authorized.

Enums§

Addressing: How a deployment maps a request to a repository.
BeginUploadResult: A replayable ticket-opening result. Tokens are retained verbatim.
Code: Error category: the 16 Connect codes, one-to-one.
Commitment: The content commitment an auth v2 signature covers.
InvalidHeader: Why a response header was refused.
MemoryFault: A one-shot failure a memory store injects, for tests of the layers above. Each store fires only the faults that apply to it.
OpKind: What an operation does, with its decoded arguments.
PresenceRequirement: Server-local §8.2 condition carried from authorization to each plan. The ref name makes an AdvanceRefs constraint apply to its head only.
Principal: The identity a request was mapped to. Non-exhaustive: M2 may add a caller-view class.
Procedure: A mkit.transport.v1.TransportService procedure.
ReplayDecision: What stage 0 does with a request, given its record.
ReplayState: Where a recorded operation stands.
StoredResult: A final result a retry gets back. There is deliberately no variant for an admission challenge or pending_verification, and a StoredRejection holds only final codes and no error detail, so neither can ever be stored.
UpdateRefResult: The result of an UpdateRef compare-and-swap.

Constants§

ADMISSION_CHALLENGE_TYPE: Fully qualified proto name of the admission challenge detail (SPEC-TRANSPORT-CONNECT §5.1).

Traits§

Clock: Injected time source, in Unix epoch milliseconds.
MaybeSend: Send on native targets; implemented for every type on wasm32.
MaybeSync: Sync on native targets; implemented for every type on wasm32.
Sleep: Injected timer, the counterpart of Clock for deadlines: Instant and tokio::time are not available on every target, so an adapter passes in its runtime’s sleep (tokio::time::sleep on native, worker::Delay on Workers) and core code bounds an external call with with_timeout.
Spawner: Injected task spawner: tokio on native, wasm_bindgen_futures or ctx.wait_until on Workers.

Functions§

classify: Classify a request with fingerprint against its record, if any.
send_wrap: Make a MaybeSend future satisfy the + Send bound that generated connectrpc service traits require. On native targets the future is already Send and is returned unchanged.
with_timeout: Run fut, giving up with Elapsed once sleep says after has passed. A future that is ready wins over an elapsed timer, and a timed-out future is dropped, so the caller must not assume its side effects did not land.

Type Aliases§

BoxFuture: A boxed future that is Send on native targets only.
BoxStream: A boxed stream that is Send on native targets only.