Crate cratefield_testing

Expand description

cratefield-testing: the conformance kit every Factory Zero module runs against (issue #9) — fake ports, an in-memory SQLite Database, and request helpers over the axum router with no network.

use cratefield_testing::{conformance, TestHarness, request};

#[test]
fn my_module_conforms() {
    conformance(Box::new(my_module::MyModule::new()));
}

#[pollster::test]
async fn join_returns_202() {
    let kit = TestHarness::new(vec![Box::new(my_module::MyModule::new())]);
    let response = request(&kit.router, http::Method::POST,
        "/v1/my-module/join", Some(r#"{"email":"nick@example.com"}"#)).await;
    assert_eq!(response.status, http::StatusCode::ACCEPTED);
}

§cratefield-testing

The conformance kit every Cratefield module runs against — public and private modules alike. Fake ports, an in-memory SQLite Database, and request helpers over the axum router with no network.

§A 15-line module test

use cratefield_testing::{conformance, request, TestHarness};

#[test]
fn my_module_conforms() {
    conformance(Box::new(my_module::MyModule::new()));
}

#[pollster::test]
async fn join_accepts_an_email() {
    let kit = TestHarness::new(vec![Box::new(my_module::MyModule::new())]);
    let res = request(&kit.router, http::Method::POST, "/v1/my-module/join",
        Some(r#"{"email":"nick@example.com"}"#)).await;
    assert_eq!(res.status, http::StatusCode::ACCEPTED);
    assert_eq!(kit.mailer.sent().len(), 1); // the confirmation mail
}

§What you get

TestHarness::new(vec![Box<dyn Module>]) — builds the harness with every port faked, applies each module’s sqlite migrations to a fresh in-memory database, assembles the router. Exposes { router, mailer, captcha, rate_limiter, db, clock, kv, http, defer, signer, events, modules, dialect }.
Parity (issue #20): TestHarness::with_database(modules, Dialect::Sqlite | Dialect::Postgres { url }) runs the same harness on a throwaway Postgres 16 database (per-harness, dropped on drop; needs the crate’s postgres feature and a server at FZ_TEST_POSTGRES_URL). TestHarness::all_dialects(make_modules) / all_dialects_with_ports(make_modules, patch) build one kit per dialect available in the environment so a module suite loops over them — one test definition, every engine. The module factory runs once per dialect: modules may carry per-build state, so kits never share instances.
Fakes: FakeMailer (records Messages; SendOk/NotConfigured/Fail modes, switchable mid-test), FakeCaptcha (allow-all or token list), FakeRateLimiter (scripted Decisions + call count), FixedClock, MemoryKeyValue, FakeHttpClient (scripted responses + captured requests), EmptyDatabase (services SELECT 1 only), FakeDefer (collects deferred futures; drain().await runs them), and a Signer with the fixed dummy TEST_HARNESS_SECRET.
request(&router, method, path, json?) -> TestResponse { status, headers, json() } — tower::ServiceExt::oneshot, no network.
conformance(module) — the shared suite, run once per available dialect: mounts + health listing, request under the prefix, migrations apply twice on fresh databases, view_for hides undeclared ports, the two-concurrent-requests request-id test (ADR 0007), and — for modules with a well_known router — that it serves at the root /.well-known and never under /v1 (#46).
full_fake_ports() — a Ports bundle with every Port wired to a fake: the one conformance’s visibility check uses.
assert_wasm_safe_deps(env!("CARGO_PKG_NAME")) — cargo tree check: no worker/wasm-bindgen/tokio/reqwest in the module’s normal dependency tree.

The in-memory Database is cratefield-adapter-sqlite; assertions on kit.db see exactly what the module wrote. On the Postgres leg kit.db is a pool on the kit’s own tokio runtime marshalled per call, so pollster-driven tests, spawned threads and deferred handlers all reach it.

MIT. Built in the open for Cratefield, a Factory Zero venture.

Modules§

vectors: The Web Push test vectors the workspace shares.

Structs§

Ask: One Classifier::ask a FakeClassifier answered, for assertions: the state as it arrived (untruncated — this fake has no provider budget to spend) and the question ids in the set, in map order.
CommentedCall: One Tracker::comment a FakeTracker accepted, for assertions.
EmptyDatabase
FakeAuth: A cratefield_core::Auth answering from a mode rather than from a key set.
FakeCaptcha
FakeClassifier: An in-memory Classifier for module tests: records every ask it answered and answers according to its ClassifierMode.
FakeCustomHostnames: An in-memory cratefield_core::CustomHostnames for module tests: records every call, walks a claim from pending to live on demand, and scripts the error the next call answers with.
FakeDefer
FakeDispatcher: An in-process Dispatcher that answers from an axum Router, so a module can be exercised through a sidecar mount without a network or a second Worker (ADR 0009). The conformance kit uses it to run the same assertions against both mounts (#64).
FakeEmbedder: A deterministic cratefield_core::Embedder for module tests: one fixed-width vector per input text, derived from the text’s own bytes, and a whitespace-word-count input_tokens.
FakeHttpClient
FakeMailer
FakePayments: An in-memory cratefield_core::Payments for module tests: records which calls were made and answers per its PaymentsMode. verify_webhook treats a signature header of "invalid" as a tampered event.
FakePrivacyProvider: An in-memory external provider. Construct with its signing secret, hold sections per subject (with_subject), and assert on what it saw (calls, applies, holds).
FakePush: An in-memory cratefield_core::Push for module tests: records every (recipient, notification) and answers according to its PushMode.
FakeRateLimiter
FakeRealtime: An in-memory cratefield_core::Realtime for module tests: records every broadcast per room and reports a fixed member list. It exercises the port a module holds (broadcast/members from outside a socket); the socket lifecycle and RoomHandler are the runtime adapter’s job, covered by the native adapter’s own tests.
FakeSidecar: A dispatcher backed by an in-process router.
FakeTextModel: An in-memory TextModel for module tests: records every Prompt it completed and answers according to its TextModelMode.
FakeTracker: An in-memory Tracker for module tests: records every accepted file/status/comment call and answers according to its TrackerMode. A call the mode answered with an error is not recorded, as FakePush does not record a failed send.
FiledCall: One Tracker::file a FakeTracker accepted, for assertions.
FixedClock
ManualClock: A Clock a test moves by hand (issue #583), for a fake whose behaviour depends on time — MemoryActors fires an alarm once the clock reaches it. Every clone shares the one instant, so the host and the test that advances it never diverge.
MemoryActors: An in-process Actors host for module tests (issue #583): one serialized actor per (kind, key), its values in memory, its one alarm driven by a ManualClock the test advances.
MemoryBlob: An in-memory cratefield_core::Blob store for module tests. MemoryBlob::new has no presigned URLs (signed_url reports Unsupported, as a directory store does); MemoryBlob::with_presign signs real SigV4 URLs that MemoryBlob::verify_presigned checks back.
MemoryKeyValue
ProviderCall: One request a FakePrivacyProvider accepted, for assertions.
StatusedCall: One Tracker::status a FakeTracker accepted, for assertions.
TempDir: A directory under the system temp directory, removed when this value is dropped — including while a panic unwinds.
TestHarness: A harness with every port faked and a migrated database, migrations applied per module on creation. The same database handle is wired into the router and exposed for assertions — SQLite in memory by default (TestHarness::new), or a throwaway Postgres 16 database for the parity suite (TestHarness::with_database, issue #20).
TestResponse: A fully-buffered test response.

Enums§

AuthMode: What a FakeAuth answers.
ClassifierMode: How a FakeClassifier answers, mirroring TextModelMode for the classifier port (issue #456).
Dialect: The database engine backing a test harness.
Fault: A way the hop can be wrong. Used by the kit’s own tests to prove the parity checks fail when the forwarder misbehaves; never by a module.
MailerMode: How a FakeMailer answers.
PaymentsCall: What a FakePayments recorded, for assertions.
PaymentsMode: How a FakePayments responds.
PushMode: How a FakePush responds, mirroring MailerMode for the push port.
TextModelMode: How a FakeTextModel answers, mirroring PushMode for the text model port (issue #429).
TrackerMode: How a FakeTracker answers, mirroring MailerMode and PushMode for the tracker port (issue #431).

Constants§

CONTRACT_ACTOR_KIND: The kind contract_actor_handler answers for and assert_actor_contract drives.
FAKE_EMBEDDER_DIMENSIONS: The width of every FakeEmbedder vector, and so of the ExactVectorIndex the kit wires in full_fake_ports — the two must agree.
FAKE_EMBEDDER_MODEL: The model name a FakeEmbedder reports.
TEST_HARNESS_SECRET
TEXT_MODEL_CONFORMANCE_CACHED_INPUT_TOKENS: The canonical cached-input count: the subset of TEXT_MODEL_CONFORMANCE_INPUT_TOKENS the scripted transport reports serving from its vendor’s prompt cache. Must stay within the input count — the suite asserts that, so the script has to satisfy it.
TEXT_MODEL_CONFORMANCE_INPUT_TOKENS: The canonical input-token count: the total prompt tokens the scripted transport reports. Every TextModel completion reports usage, so the suite asserts this count exactly — an adapter whose vendor omits a usage block fails here, not silently in a caller’s cost accounting.
TEXT_MODEL_CONFORMANCE_OUTPUT_TOKENS: The canonical output-token count the scripted transport reports.
TEXT_MODEL_CONFORMANCE_REPLY: The canonical reply the conformance suite expects, verbatim: short and fixed, so an adapter test can script its transport to answer exactly this in its vendor’s wire shape.

Functions§

assert_actor_contract: Proves actors keeps the Actors contract, using advance to move the host’s clock forward by a Duration so a pending alarm fires — an in-process host advances its own clock, an adapter’s test drives whatever its runtime exposes.
assert_batch_is_atomic: Proves db’s batch_atomic is all-or-nothing: a committed batch is fully visible, a failing batch is invisible. Runs its own probe table, so it can be called against any adapter with a writable connection.
assert_blob_large_round_trips: Proves blob satisfies the large-object half of the Blob port (issue #586): streamed puts and gets for an object past MAX_BLOB_BYTES, the declared ceiling enforced mid-stream, multipart uploads that assemble in part order, abort leaving nothing behind, the bad-key rules, and a listing that never surfaces another module’s keys.
assert_blob_round_trips: Proves db round-trips bound bytes exactly: a Vec<u8> bound as Value::Bytes reads back byte-for-byte identical, and a NULL blob reads back as no bytes. Runs its own probe table, so it can be called against any adapter with a writable connection.
assert_wasm_safe_deps: Asserts the named crate’s normal dependency tree carries none of the dependencies a wasm module must never see: worker, wasm-bindgen, tokio, reqwest. Shell-out check (cargo tree --edges normal) so the assertion is made against the graph cargo actually resolves, not a copy of it that could drift.
classifier_conformance: Asserts the Classifier trait contract against a classifier whose transport is already scripted to answer classifier_conformance_questions about classifier_conformance_state. The suite makes exactly one ask call — script that one request and no other.
classifier_conformance_questions: The canonical question set the conformance suite asks: one Question::Choice, one Question::Score, one Question::Noul, so every shape of the port is exercised once. The ids are fixed — topic, severity, urgent — and an adapter test scripts its transport to answer exactly these before running the suite.
classifier_conformance_state: The state the conformance suite asks about: short and fixed, so a scripted transport can match it.
classifier_not_configured: Asserts an adapter with no key, binding or model behind it answers ClassifierError::NotConfigured rather than panicking — the unwired port is an error the caller matches, so a module can degrade on it.
classifier_rejects_malformed_questions: Asserts the contract that needs no transport: a malformed question set — empty, blank id, blank instructions, a one-criterion choice, a one-level score — comes back as ClassifierError::Rejected and never panics. Safe to run against an unconfigured adapter: the set is refused before any provider is reached.
classifier_truncates_long_state: Asserts that asking with a state longer than profile().max_state_chars still succeeds — truncation is the adapter’s job, so an over-limit state must never surface as an error to the caller.
conformance: Runs the shared conformance suite against one module, once per dialect available in the environment (issue #20 — SQLite always, Postgres when FZ_TEST_POSTGRES_URL names a server):
conformance_in_process_only: conformance without the sidecar parity axis (issue #64). reason is recorded in code and printed by the run, so a module that opts out says why in the same place it opts out. Use it only for a module that genuinely cannot be sidecar-mounted; “it fails” is not a reason.
contract_actor_handler: The contract handler, for a host that takes one Arc.
contract_actor_handlers: A registry holding just the contract handler under CONTRACT_ACTOR_KIND, for a host built from ActorHandlers.
error_json: A Colonizer-shaped error body, {"error": message} under application/json.
full_fake_ports: A Ports bundle with every Port wired to a fake, for checks that need a full bundle: conformance’s visibility check (declared ports must survive Ports::view_for, undeclared ones must be hidden), and a runtime proving it carries every port through a copy of its bundle. dispatcher and tenants are not ports and stay None.
problem_json: An RFC 9457 application/problem+json response — the shape Owlpost answers errors with: type, title, status and detail, the status both the HTTP status and the body’s own field. Build one for a scripted sequence, or take it from a preset like FakeHttpClient::owlpost_problem.
push_recipient_conformance: Runs a Push adapter against every Recipient variant and asserts the port’s contract for transports it does not serve (issue #177): a clean PushError::Rejected naming the recipient as unsupported — never a panic, never a silent success, and never a Transient that an outbox would retry forever.
request: Sends a request through the router without a network. json (when Some) becomes a JSON body with content-type: application/json.
request_as: The same, carrying Authorization: Bearer <token>.
request_chunks: Sends a request whose body is the given chunks, as a chunked (no content-length) transfer — what a client uploading a stream produces, and the only shape that exercises a route’s cratefield_core::RequestStream mid-stream ceiling rather than the declared-length pre-check. content-type is set to application/octet-stream.
shared_sidecar: Wraps a FakeSidecar so the same handle can be given to Ports and still be asked how many calls it saw.
sidecar_parity: Asserts that a module answers identically in-process and behind a sidecar mount (issue #64, ADR 0009: a caller cannot tell which).
sign_provider_scheme: The headers a configured ProviderScheme reads for body signed at t: the signature header carrying one entry — the scheme’s prefix then the hex or base64 of HMAC-SHA256(secret, payload) — and, when the scheme names a timestamp header, that header set to t. The payload is "{t}.{body}" when the scheme carries a timestamp and the raw body otherwise; a scheme with no timestamp header ignores t.
sign_stripe_style: The delivery headers Stripe sends for body signed at t: one header named header, holding t=<t>,v1=<hex>, the hex being HMAC-SHA256(secret, "{t}.{body}"). The exact inverse of StripeStyle’s extraction — feed the result straight to WebhookVerifier::verify with now_unix == t.
text_model_conformance: Asserts the TextModel trait contract against a text model whose transport is already scripted to complete text_model_conformance_prompt with TEXT_MODEL_CONFORMANCE_REPLY and the canonical counts in its vendor’s wire shape. The suite makes exactly one complete call — script that one request and no other.
text_model_conformance_not_configured: Asserts an adapter with no key behind it answers TextModelError::NotConfigured rather than panicking — the unwired port is an error the caller matches, so a module can degrade on it.
text_model_conformance_prompt: The canonical prompt the suite asks: one user turn, no system prompt, no schema, the default token ceiling. An adapter test scripts its transport to answer TEXT_MODEL_CONFORMANCE_REPLY with the canonical counts (TEXT_MODEL_CONFORMANCE_INPUT_TOKENS, TEXT_MODEL_CONFORMANCE_OUTPUT_TOKENS, TEXT_MODEL_CONFORMANCE_CACHED_INPUT_TOKENS) for this prompt.
text_model_image_bounds_conformance: Asserts a TextModel adapter refuses an over-limit image prompt (issue #628) before any network call: too many images, one image over MAX_IMAGE_ENCODED_BYTES, or images together over MAX_PROMPT_IMAGE_ENCODED_BYTES, each refused with the matching TextModelError::ImageLimit.
vector_index_conformance: Asserts the VectorIndex contract against an adapter, over small three-dimensional vectors (issue #561):
with_retry_after: Adds Retry-After: <retry_after_secs> to a response in seconds — the header a 429 carries and the adapters read as a retry hint.