Crate cratefield_testing

Expand description

cratefield-testing: the conformance kit every Factory Zero module runs against (issue #9) — fake ports, an in-memory SQLite Database, and request helpers over the axum router with no network.

use cratefield_testing::{conformance, TestHarness, request};

#[test]
fn my_module_conforms() {
    conformance(Box::new(my_module::MyModule::new()));
}

#[pollster::test]
async fn join_returns_202() {
    let kit = TestHarness::new(vec![Box::new(my_module::MyModule::new())]);
    let response = request(&kit.router, http::Method::POST,
        "/v1/my-module/join", Some(r#"{"email":"nick@example.com"}"#)).await;
    assert_eq!(response.status, http::StatusCode::ACCEPTED);
}

§cratefield-testing

The conformance kit every Cratefield module runs against — public and private modules alike. Fake ports, an in-memory SQLite Database, and request helpers over the axum router with no network.

§A 15-line module test

use cratefield_testing::{conformance, request, TestHarness};

#[test]
fn my_module_conforms() {
    conformance(Box::new(my_module::MyModule::new()));
}

#[pollster::test]
async fn join_accepts_an_email() {
    let kit = TestHarness::new(vec![Box::new(my_module::MyModule::new())]);
    let res = request(&kit.router, http::Method::POST, "/v1/my-module/join",
        Some(r#"{"email":"nick@example.com"}"#)).await;
    assert_eq!(res.status, http::StatusCode::ACCEPTED);
    assert_eq!(kit.mailer.sent().len(), 1); // the confirmation mail
}

§What you get

TestHarness::new(vec![Box<dyn Module>]) — builds the harness with every port faked, applies each module’s sqlite migrations to a fresh in-memory database, assembles the router. Exposes { router, mailer, captcha, rate_limiter, db, clock, kv, http, defer, signer, events, modules, dialect }.
Parity (issue #20): TestHarness::with_database(modules, Dialect::Sqlite | Dialect::Postgres { url }) runs the same harness on a throwaway Postgres 16 database (per-harness, dropped on drop; needs the crate’s postgres feature and a server at FZ_TEST_POSTGRES_URL). TestHarness::all_dialects(make_modules) / all_dialects_with_ports(make_modules, patch) build one kit per dialect available in the environment so a module suite loops over them — one test definition, every engine. The module factory runs once per dialect: modules may carry per-build state, so kits never share instances.
Fakes: FakeMailer (records Messages; SendOk/NotConfigured/Fail modes, switchable mid-test), FakeCaptcha (allow-all or token list), FakeRateLimiter (scripted Decisions + call count), FixedClock, MemoryKeyValue, FakeHttpClient (scripted responses + captured requests), EmptyDatabase (services SELECT 1 only), FakeDefer (collects deferred futures; drain().await runs them), and a Signer with the fixed dummy TEST_HARNESS_SECRET.
request(&router, method, path, json?) -> TestResponse { status, headers, json() } — tower::ServiceExt::oneshot, no network.
conformance(module) — the shared suite, run once per available dialect: mounts + health listing, request under the prefix, migrations apply twice on fresh databases, view_for hides undeclared ports, the two-concurrent-requests request-id test (ADR 0007), and — for modules with a well_known router — that it serves at the root /.well-known and never under /v1 (#46).
full_fake_ports() — a Ports bundle with every Port wired to a fake: the one conformance’s visibility check uses.
assert_wasm_safe_deps(env!("CARGO_PKG_NAME")) — cargo tree check: no worker/wasm-bindgen/tokio/reqwest in the module’s normal dependency tree.

The in-memory Database is cratefield-adapter-sqlite; assertions on kit.db see exactly what the module wrote. On the Postgres leg kit.db is a pool on the kit’s own tokio runtime marshalled per call, so pollster-driven tests, spawned threads and deferred handlers all reach it.

MIT. Built in the open for Cratefield, a Factory Zero venture.

Modules§

vectors: The Web Push test vectors the workspace shares.

Structs§

Ask: One Classifier::ask a FakeClassifier answered, for assertions: the state as it arrived (untruncated — this fake has no provider budget to spend) and the question ids in the set, in map order.
CommentedCall: One Tracker::comment a FakeTracker accepted, for assertions.
EmptyDatabase
FakeAuth: A cratefield_core::Auth answering from a mode rather than from a key set.
FakeCaptcha
FakeClassifier: An in-memory Classifier for module tests: records every ask it answered and answers according to its ClassifierMode.
FakeCustomHostnames: An in-memory cratefield_core::CustomHostnames for module tests: records every call, walks a claim from pending to live on demand, and scripts the error the next call answers with.
FakeDefer
FakeDispatcher: An in-process Dispatcher that answers from an axum Router, so a module can be exercised through a sidecar mount without a network or a second Worker (ADR 0009). The conformance kit uses it to run the same assertions against both mounts (#64).
FakeEmbedder: A deterministic cratefield_core::Embedder for module tests: one fixed-width vector per input text, derived from the text’s own bytes, and a whitespace-word-count input_tokens.
FakeHttpClient
FakeMailer
FakePayments: An in-memory cratefield_core::Payments for module tests: records which calls were made and answers per its PaymentsMode. verify_webhook treats a signature header of "invalid" as a tampered event.
FakePrivacyProvider: An in-memory external provider. Construct with its signing secret, hold sections per subject (with_subject), and assert on what it saw (calls, applies, holds).
FakePush: An in-memory cratefield_core::Push for module tests: records every (recipient, notification) and answers according to its PushMode.
FakeRateLimiter
FakeRealtime: An in-memory cratefield_core::Realtime for module tests: records every broadcast per room and reports a fixed member list. It exercises the port a module holds (broadcast/members from outside a socket); the socket lifecycle and RoomHandler are the runtime adapter’s job, covered by the native adapter’s own tests.
FakeSidecar: A dispatcher backed by an in-process router.
FakeTextModel: An in-memory TextModel for module tests: records every Prompt it completed and answers according to its TextModelMode.
FakeTracker: An in-memory Tracker for module tests: records every accepted file/status/comment call and answers according to its TrackerMode. A call the mode answered with an error is not recorded, as FakePush does not record a failed send.
FakeUrlSafety: A UrlSafety that knows every URL as clean. It runs the port’s own check_url first, as every adapter must, so a fixture URL that is not http(s) or is over the length ceiling is refused the same way the real adapters refuse it. Wired into crate::full_fake_ports so the conformance kit and the runtime clone checks see the port present.
FiledCall: One Tracker::file a FakeTracker accepted, for assertions.
FixedClock
ManualClock: A Clock a test moves by hand (issue #583), for a fake whose behaviour depends on time — MemoryActors fires an alarm once the clock reaches it. Every clone shares the one instant, so the host and the test that advances it never diverge.
MemoryActors: An in-process Actors host for module tests (issue #583): one serialized actor per (kind, key), its values in memory, its one alarm driven by a ManualClock the test advances.
MemoryBlob: An in-memory cratefield_core::Blob store for module tests. MemoryBlob::new has no presigned URLs (signed_url reports Unsupported, as a directory store does); MemoryBlob::with_presign signs real SigV4 URLs that MemoryBlob::verify_presigned checks back.
MemoryKeyValue
ProviderCall: One request a FakePrivacyProvider accepted, for assertions.
StatusedCall: One Tracker::status a FakeTracker accepted, for assertions.
TempDir: A directory under the system temp directory, removed when this value is dropped — including while a panic unwinds.
TestHarness: A harness with every port faked and a migrated database, migrations applied per module on creation. The same database handle is wired into the router and exposed for assertions — SQLite in memory by default (TestHarness::new), or a throwaway Postgres 16 database for the parity suite (TestHarness::with_database, issue #20).
TestResponse: A fully-buffered test response.

Enums§

AuthMode: What a FakeAuth answers.
ClassifierMode: How a FakeClassifier answers, mirroring TextModelMode for the classifier port (issue #456).
Dialect: The database engine backing a test harness.
Fault: A way the hop can be wrong. Used by the kit’s own tests to prove the parity checks fail when the forwarder misbehaves; never by a module.
MailerMode: How a FakeMailer answers.
PaymentsCall: What a FakePayments recorded, for assertions.
PaymentsMode: How a FakePayments responds.
PushMode: How a FakePush responds, mirroring MailerMode for the push port.
TextModelMode: How a FakeTextModel answers, mirroring PushMode for the text model port (issue #429).
TrackerMode: How a FakeTracker answers, mirroring MailerMode and PushMode for the tracker port (issue #431).

Constants§

CONTRACT_ACTOR_KIND: The kind contract_actor_handler answers for and assert_actor_contract drives.
FAKE_EMBEDDER_DIMENSIONS: The width of every FakeEmbedder vector, and so of the ExactVectorIndex the kit wires in full_fake_ports — the two must agree.
FAKE_EMBEDDER_MODEL: The model name a FakeEmbedder reports.
SPEECH_CONFORMANCE_AUDIO_BYTES: The canonical audio the scripted synthesis returns, byte for byte: a short mp3-shaped stand-in the suite asserts is carried through unharmed, not that it decodes.
SPEECH_CONFORMANCE_AUDIO_MILLIS: The canonical length of the transcribed audio, in milliseconds — what the canonical WAV clip works out to, and what the scripted transcript must report as SpeechUsage::audio_millis.
SPEECH_CONFORMANCE_SYNTH_TEXT: The canonical text the suite synthesizes.
SPEECH_CONFORMANCE_TRANSCRIPT_TEXT: The canonical transcript the suite expects, verbatim: four words, short and fixed, so an adapter test scripts its transport to answer exactly this in its vendor’s wire shape.
SPEECH_CONFORMANCE_WORD_TIMINGS: The canonical word timings as (word, start_ms, end_ms): the four words of SPEECH_CONFORMANCE_TRANSCRIPT_TEXT, the first starting at zero.
TEST_HARNESS_SECRET
TEXT_MODEL_CONFORMANCE_CACHED_INPUT_TOKENS: The canonical cached-input count: the subset of TEXT_MODEL_CONFORMANCE_INPUT_TOKENS the scripted transport reports serving from its vendor’s prompt cache. Must stay within the input count — the suite asserts that, so the script has to satisfy it.
TEXT_MODEL_CONFORMANCE_INPUT_TOKENS: The canonical input-token count: the total prompt tokens the scripted transport reports. Every TextModel completion reports usage, so the suite asserts this count exactly — an adapter whose vendor omits a usage block fails here, not silently in a caller’s cost accounting.
TEXT_MODEL_CONFORMANCE_OUTPUT_TOKENS: The canonical output-token count the scripted transport reports.
TEXT_MODEL_CONFORMANCE_REPLY: The canonical reply the conformance suite expects, verbatim: short and fixed, so an adapter test can script its transport to answer exactly this in its vendor’s wire shape.

Functions§

assert_actor_contract: Proves actors keeps the Actors contract, using advance to move the host’s clock forward by a Duration so a pending alarm fires — an in-process host advances its own clock, an adapter’s test drives whatever its runtime exposes.
assert_batch_is_atomic: Proves db’s batch_atomic is all-or-nothing: a committed batch is fully visible, a failing batch is invisible. Runs its own probe table, so it can be called against any adapter with a writable connection.
assert_blob_large_round_trips: Proves blob satisfies the large-object half of the Blob port (issue #586): streamed puts and gets for an object past MAX_BLOB_BYTES, the declared ceiling enforced mid-stream, multipart uploads that assemble in part order, abort leaving nothing behind, the bad-key rules, and a listing that never surfaces another module’s keys.
assert_blob_round_trips: Proves db round-trips bound bytes exactly: a Vec<u8> bound as Value::Bytes reads back byte-for-byte identical, and a NULL blob reads back as no bytes. Runs its own probe table, so it can be called against any adapter with a writable connection.
assert_budget_daily_quota: Proves db can carry the daily-quota half of the outbound budgets (issue #765): a daily_limit of 10 at cost 1 fires EVENT_QUOTA_WARNING exactly once on the eighth send (the 80% mark) and EVENT_QUOTA_EXHAUSTED exactly once on the tenth — at-most-once, with no bookkeeping — and the eleventh send is refused with HttpError::BudgetExhausted whose retry_after reaches the UTC day boundary, past which the quota is fresh again. A budget with daily_limit: None never refuses on quota and never alerts.
assert_budget_pacing: Proves db can carry the pacing half of the outbound budgets (issue #765): capacity 2 refilled at 1 token per second, cost 1 per request — two sends spend the whole bucket, the third paces (the recorded sleeper waits no longer than the policy’s max_delay) or refuses with HttpError::BudgetExhausted naming a positive retry_after, and one refill window later the request goes through. A second key spends its own full bucket and holds its own row: keys do not share a bucket.
assert_budget_passthrough: Proves the pass-through half of the outbound budgets (issue #765): a router that answers None governs nothing — the request reaches the inner client verbatim (method, URI, headers, body and the request extensions an Idempotent marker rides in), the response passes back untouched, and the budget store stays empty because nothing was spent: no bucket row, no quota row, no sleeper wait.
assert_budget_retries: Proves the retry half of the outbound budgets (issue #765): a 429 stating Retry-After: 7 on an idempotent GET is honoured verbatim — the recorded sleeper waits exactly 7 s and the exchange lands on the second attempt — the same script on a POST is not a retry (the 429 itself is the answer), a POST the caller vouches for with the Idempotent extension set to true retries, and a transport failure on a GET retries on the policy’s exponential schedule (its own base, doubled).
assert_wasm_safe_deps: Asserts the named crate’s normal dependency tree carries none of the dependencies a wasm module must never see: worker, wasm-bindgen, tokio, reqwest. Shell-out check (cargo tree --edges normal) so the assertion is made against the graph cargo actually resolves, not a copy of it that could drift.
classifier_conformance: Asserts the Classifier trait contract against a classifier whose transport is already scripted to answer classifier_conformance_questions about classifier_conformance_state. The suite makes exactly one ask call — script that one request and no other.
classifier_conformance_questions: The canonical question set the conformance suite asks: one Question::Choice, one Question::Score, one Question::Noul, so every shape of the port is exercised once. The ids are fixed — topic, severity, urgent — and an adapter test scripts its transport to answer exactly these before running the suite.
classifier_conformance_state: The state the conformance suite asks about: short and fixed, so a scripted transport can match it.
classifier_not_configured: Asserts an adapter with no key, binding or model behind it answers ClassifierError::NotConfigured rather than panicking — the unwired port is an error the caller matches, so a module can degrade on it.
classifier_rejects_malformed_questions: Asserts the contract that needs no transport: a malformed question set — empty, blank id, blank instructions, a one-criterion choice, a one-level score — comes back as ClassifierError::Rejected and never panics. Safe to run against an unconfigured adapter: the set is refused before any provider is reached.
classifier_truncates_long_state: Asserts that asking with a state longer than profile().max_state_chars still succeeds — truncation is the adapter’s job, so an over-limit state must never surface as an error to the caller.
conformance: Runs the shared conformance suite against one module, once per dialect available in the environment (issue #20 — SQLite always, Postgres when FZ_TEST_POSTGRES_URL names a server):
conformance_in_process_only: conformance without the sidecar parity axis (issue #64). reason is recorded in code and printed by the run, so a module that opts out says why in the same place it opts out. Use it only for a module that genuinely cannot be sidecar-mounted; “it fails” is not a reason.
contract_actor_handler: The contract handler, for a host that takes one Arc.
contract_actor_handlers: A registry holding just the contract handler under CONTRACT_ACTOR_KIND, for a host built from ActorHandlers.
error_json: A Colonizer-shaped error body, {"error": message} under application/json.
full_fake_ports: A Ports bundle with every Port wired to a fake, for checks that need a full bundle: conformance’s visibility check (declared ports must survive Ports::view_for, undeclared ones must be hidden), and a runtime proving it carries every port through a copy of its bundle. dispatcher and tenants are not ports and stay None.
problem_json: An RFC 9457 application/problem+json response — the shape Owlpost answers errors with: type, title, status and detail, the status both the HTTP status and the body’s own field. Build one for a scripted sequence, or take it from a preset like FakeHttpClient::owlpost_problem.
push_recipient_conformance: Runs a Push adapter against every Recipient variant and asserts the port’s contract for transports it does not serve (issue #177): a clean PushError::Rejected naming the recipient as unsupported — never a panic, never a silent success, and never a Transient that an outbox would retry forever.
request: Sends a request through the router without a network. json (when Some) becomes a JSON body with content-type: application/json.
request_as: The same, carrying Authorization: Bearer <token>.
request_chunks: Sends a request whose body is the given chunks, as a chunked (no content-length) transfer — what a client uploading a stream produces, and the only shape that exercises a route’s cratefield_core::RequestStream mid-stream ceiling rather than the declared-length pre-check. content-type is set to application/octet-stream.
shared_sidecar: Wraps a FakeSidecar so the same handle can be given to Ports and still be asked how many calls it saw.
sidecar_parity: Asserts that a module answers identically in-process and behind a sidecar mount (issue #64, ADR 0009: a caller cannot tell which).
sign_provider_scheme: The headers a configured ProviderScheme reads for body signed at t: the signature header carrying one entry — the scheme’s prefix then the hex or base64 of HMAC-SHA256(secret, payload) — and, when the scheme names a timestamp header, that header set to t. The payload is "{t}.{body}" when the scheme carries a timestamp and the raw body otherwise; a scheme with no timestamp header ignores t.
sign_stripe_style: The delivery headers Stripe sends for body signed at t: one header named header, holding t=<t>,v1=<hex>, the hex being HMAC-SHA256(secret, "{t}.{body}"). The exact inverse of StripeStyle’s extraction — feed the result straight to WebhookVerifier::verify with now_unix == t.
speech_conformance: Asserts the Speech trait contract against an adapter whose transport is scripted to answer the canonical fixture: the speech_conformance_transcribe_request clip with SPEECH_CONFORMANCE_TRANSCRIPT_TEXT (usage SPEECH_CONFORMANCE_AUDIO_MILLIS), a synthesis of SPEECH_CONFORMANCE_SYNTH_TEXT in Mp3 streaming out SPEECH_CONFORMANCE_AUDIO_BYTES byte for byte, and a non-empty voice list.
speech_conformance_not_configured: Asserts an adapter with no key behind it answers SpeechError::NotConfigured on every operation — the unwired port is an error the caller matches.
speech_conformance_transcribe_request: A small valid WAV clip (8 000 bytes/second, 12 000 data bytes — exactly SPEECH_CONFORMANCE_AUDIO_MILLIS) with no declared duration, so the length derives from the RIFF header. The request a fixture scripts against: audio/wav, language hint en.
text_model_conformance: Asserts the TextModel trait contract against a text model whose transport is already scripted to complete text_model_conformance_prompt with TEXT_MODEL_CONFORMANCE_REPLY and the canonical counts in its vendor’s wire shape. The suite makes exactly one complete call — script that one request and no other.
text_model_conformance_not_configured: Asserts an adapter with no key behind it answers TextModelError::NotConfigured rather than panicking — the unwired port is an error the caller matches, so a module can degrade on it.
text_model_conformance_prompt: The canonical prompt the suite asks: one user turn, no system prompt, no schema, the default token ceiling. An adapter test scripts its transport to answer TEXT_MODEL_CONFORMANCE_REPLY with the canonical counts (TEXT_MODEL_CONFORMANCE_INPUT_TOKENS, TEXT_MODEL_CONFORMANCE_OUTPUT_TOKENS, TEXT_MODEL_CONFORMANCE_CACHED_INPUT_TOKENS) for this prompt.
text_model_image_bounds_conformance: Asserts a TextModel adapter refuses an over-limit image prompt (issue #628) before any network call: too many images, one image over MAX_IMAGE_ENCODED_BYTES, or images together over MAX_PROMPT_IMAGE_ENCODED_BYTES, each refused with the matching TextModelError::ImageLimit.
vector_index_conformance: Asserts the VectorIndex contract against an adapter, over small three-dimensional vectors (issue #561):
with_retry_after: Adds Retry-After: <retry_after_secs> to a response in seconds — the header a 429 carries and the adapters read as a retry hint.