rightkit-qa 0.2.12

Rust QA harness for Right Suite apps: engine black-box scenarios, hidden native UI driving through rightkit-control, Rust-test scenario wrapper, dynamic paid-provider mocks, loudness/WAV/PNG/frame validators, run lock, orphan sweep, hashed evidence.
Documentation

rightkit-qa

Rust QA harness for Right Suite applications. It provides declarative scenarios, engine and app control, run locking, isolated profiles, process handling, and hashed evidence.

The browser feature enables Chrome targets through rightkit-browser.

Run storage

Run roots, harness evidence, captures & summaries remain under <managed>/runs/<app>/<run-id>. App homes, profiles, service/port files, live logs & app-provided output paths use an owned internal system-temp companion recorded as appRoot in .rightkit-run.json. On macOS, companions always use /private/tmp/rightkit-qa/<app-prefix>-<random> with short names so launchd can open app files & control socket paths fit sockaddr_un.

Finish copies app logs to managed evidence/, app-written evidence to evidence/app/ & app captures to captures/ before releasing ownership. Disposal removes temp companions; failed copies or cleanup preserve ownership metadata for recovery. Retention & orphan pruning validate companion bindings & protect live runs.

Rust integration tests

harness::Harness::scenario(name, tier, requires, |sc| ...) wraps a #[test] body with tier gating, requires-skip, a per-scenario evidence directory (<evidence root>/<slug>/evidence.json, hashed artifacts), an orphan-process check and a summary.jsonl skip ledger. A skipped scenario prints [qa] SKIP name: reason, because libtest reports it as ok.

Settings under rightkit cargo

rightkit cargo test starts the test binary with a scrubbed environment and refuses nested managed builds. Do not pass knobs as environment variables. Use this pattern instead:

  1. Declare tiers as cargo features of the test crate (full = [], paid = []), and pass them with Harness::with_feature_tiers(&[("full", cfg!(feature = "full")), ("paid", cfg!(feature = "paid"))]). Run them with rightkit cargo test -p my-e2e --features full,paid. The fast tier always runs.
  2. Put knobs in a KEY=VALUE settings file and read them through settings::Settings (get, require, get_bool, get_u64, get_f64, get_list, get_path). requires = ["env:KEY"] is answered from the same file.
  3. Build binaries under test first with a separate rightkit cargo build -p <bin>, then locate them next to the test executable's deps/ directory.

Search order (the first existing file wins; files are not merged):

  1. RIGHTKIT_QA_SETTINGS (a path; an error if it is set and the file is missing)
  2. <root>/.cache/qa/settings.env
  3. <root>/.cache/e2e/settings.env
  4. <root>/rightkit-qa.settings.env

A non-empty process environment variable overrides the file for a single key. This only applies to direct, unmanaged settings. The harness reads RIGHTKIT_QA_TIER to override tier selection (Harness::with_tier_key renames it); legacy RIGHTKIT_QA_EVIDENCE is accepted by settings for compatibility but managed harness runs always write evidence under their managed run root.

UI scenarios against a real app (macOS)

Scenarios that launch a real .app (control::launch) must run from the user's login session. Under rightkit cargo test, the test binary runs in the build broker's context, where open has no GUI session and exits 1. Build through the broker and run the binary yourself; scripts/run-ui-tests.sh does both and keeps your shell's environment:

tools/rightkit/scripts/run-ui-tests.sh -p cutright-e2e --test ui -- --nocapture

On macOS, control::launch stages raw binaries & supplied .app bundles under <appRoot>/apps/<launch-id>/, then passes every harness-owned home/temp/profile path through open --env NAME=VALUE. HOME/USERPROFILE point to appRoot; data/profile keys point to appRoot/data; TMPDIR/TMP/TEMP point to appRoot/tmp. RIGHTKIT_SUITE_ROOT points to appRoot/suite-<launch-id-prefix> (including runtime/<service>/service.json); live stdout/stderr use appRoot/app-<launch-id-prefix>.log. The launcher cwd & harness scratch directories also use appRoot.

Caller-supplied LaunchSpec::env still rejects credential keys & allows only RIGHTKIT_* keys on macOS. Absolute caller paths must live inside appRoot; callers cannot replace suite/service identity or move isolation paths outside it. ${data_dir} & ${evidence_dir} scenario templates use internal companion paths; artifacts are copied into managed evidence before hashing so disposal cannot invalidate them. App-specific QA knobs should use RIGHTKIT_<APP>_* keys honoured only in debug QA builds.

Hidden launches: the app must not take focus itself. When RIGHTKIT_QA_HIDDEN=1, skip any set_focus()/show() on startup, and build with activate_ignoring_other_apps(false) on macOS (see rightkit-control/examples/control-test-app/src/main.rs). Compare paths with std::fs::canonicalize: the QA data dir lives under /var/folders, which macOS resolves to /private/var/folders.

Control::wait_for_text(Some("#status"), "Ready", timeout) waits for visible text (None searches the whole body).

Mocks and validators

  • mock::MockServer::start_with(|req, base| Reply) serves dynamic, stateful and binary (Reply::bytes) responses for paid-provider contracts. MockServer::start(routes) keeps the static JSON routes; MockServer::start_static(vec![(method, path, Reply)]) serves static binary replies.
  • Every 200 reply to GET/HEAD honours a single HTTP Range (bytes=a-b, a-, suffix -n): 206 with Content-Range, 416 when unsatisfiable, the full 200 when Range is absent, malformed, multi-range or its If-Range does not match the reply's ETag. Reply::ignore_range() models a server without resume; Reply::cut_after(n) drops the connection after n body bytes.
  • loudness: pure-Rust ITU-R BS.1770-4 integrated loudness, EBU loudness range, sample peak and true peak. It is checked against ffmpeg ebur128 to within 0.2 LU.
  • media: WAV decode and stats, PNG non-trivial (pure Rust), and decoded-frame non-uniformity. ffmpeg is only needed to decode compressed audio and video, and its binary path is always passed in.

The browser target reaches a local dev server only through browser::allow_loopback_origin / BrowserTarget::launch_for_origins. That allowance is scoped to one scheme, host and port.

Publish through the RightKit release workflow.