llm-browser-testkit

Describe browser tests in plain English. The LLM figures out which elements to click and whether the page looks right — plus A2A agents, MCP tool-calling, cost tracking, and budgets.

llm-browser-testkit run smoke.toml

Quick start
Write your first test
Step reference
Assertion presets
CLI reference
Endpoints
A2A agents
Run as an A2A agent
MCP tools
MCP server
Cost tracking & budgets
How it works
Use as a library
LLM authentication
License

Quick start

Everything you need for a first green run — copy-paste, no prior setup:

# 1. Install the CLI
cargo install llm-browser-testkit

# 2. Point it at any OpenAI-compatible API
export HARNESS_LLM_TEST_URL=https://api.openai.com
export HARNESS_LLM_TEST_MODEL=gpt-4o-mini
export HARNESS_LLM_API_KEY=sk-...

# 3. Describe one test in a tiny TOML file (example.com — no account needed)
cat > hello.toml <<'EOF'
[config]
base_url = "https://example.com"
start_url = "/"

[[test]]
name = "Homepage loads"

[[test.steps]]
kind = "navigate"
url = "/"

[[test.steps]]
kind = "assert"
preset = "no_error_on_page"
EOF

# 4. Run it — Chrome runs headless, the LLM checks the page
llm-browser-testkit run hello.toml

Example output:

Test: Homepage loads — passed (6.2s, $0.0005, 138 tokens, 1 calls, 2+0+0 steps)
run passed: tests 1 passed, 0 failed | steps 2 passed, 0 failed, 0 skipped | $0.0005 | 138 tokens | 1 calls

Write your first test

The quick-start hello.toml is the smallest useful scenario. Its three blocks:

[config] — base_url is the app under test; start_url is where Chrome loads first.
[[test]] — one named test, built from [[test.steps]] that run top to bottom.
Steps — every step has a kind: navigate opens a URL relative to base_url; assert sends the page to the LLM and expects PASS/FAIL. preset picks a built-in check (no_error_on_page = no errors, stack traces, or broken UI).

Reusable checks go into [[definitions]] — name a preset or prompt once and reference it from any assertion:

[[definitions]]
name = "no_errors"
preset = "no_error_on_page"

[[definitions]]
name = "example_domain_visible"
preset = "text_visible"
assert_text = "Example Domain"

[[test]]
name = "Homepage loads"

[[test.steps]]
kind = "navigate"
url = "/"

[[test.steps]]
kind = "assert"
definition = "no_errors"

[[test.steps]]
kind = "assert"
definition = "example_domain_visible"

Run it:

llm-browser-testkit run hello.toml

Step reference

Every step has a kind. Required fields depend on the kind.

`kind`	What it does	Required	Optional
`navigate`	Open a URL	`url`	`wait_after_ms`
`click`	Click an element	`target`	`selector`, `wait_after_ms`, `endpoint`, `idempotent`
`type`	Type into a field	`target`, `text`	`selector`, `wait_after_ms`, `endpoint`, `idempotent`
`wait`	Wait for an element and/or visible text	`target`	`selector`, `text`, `timeout_ms`, `endpoint`, `idempotent`
`assert`	Check the page	one of `definition`, `preset`, or `prompt`	`assert_text`, `endpoint`
`screenshot`	Save a .png	—	`path`
`agent`	Call an A2A agent	`agent`, `task`	`definition`
`mcp`	Call an MCP tool	`server`, `tool`	`args`

target is natural language ("the submit button", "the search input"). The LLM looks at the page DOM and picks the right CSS selector at runtime. Skip the LLM with an explicit selector.

idempotent — an optional flag on click, type and wait steps. When the step's target is absent, the step is reported skipped instead of failed: the action was already done or not applicable. This is the generic building block for flows that repeat in one browser session — e.g. logging in on every viewport-matrix variant of the same test:

[[test.steps]]
kind = "navigate"
url = "/auth/login"

# Already authenticated? The form is gone, so these steps skip
# instead of failing.
[[test.steps]]
kind = "type"
selector = "#email"
target = "the email input"
text = "admin@example.com"
idempotent = true

[[test.steps]]
kind = "type"
selector = "#password"
target = "the password input"
text = "correct horse battery staple"
idempotent = true

[[test.steps]]
kind = "wait"
selector = "input[name="cf-turnstile-response"][value]:not([value=""])"
target = "the bot-protection token"
timeout_ms = 15000
idempotent = true

[[test.steps]]
kind = "click"
selector = "button.btn--landing.btn--primary"
target = "the sign-in button"
idempotent = true

# The final check stays strict: a real login attempt that never
# reaches the authenticated shell still fails the test.
[[test.steps]]
kind = "wait"
selector = "app-account-shell"
target = "the authenticated shell"
timeout_ms = 30000

Semantics:

idempotent click / type: probe for up to 5s; absent target → skipped.
idempotent wait: run the wait as normal; a timeout → skipped instead of failed.
Skipped steps do not fail the test and do not trigger fail-fast.
Keep the final verification step strict (no idempotent) so real failures in the middle of an idempotent flow still surface.

endpoint routes this step to a specific endpoint. Use it to send element targeting to one model and assertions to another.

wait with text waits until the page's visible text contains a substring — no selector or LLM needed:

[[test.steps]]
kind = "wait"
target = "the success message"
text = "Welcome back"
timeout_ms = 5000

Set both selector and text to require both conditions. The combined wait shares one timeout_ms budget.

Failure diagnostics & artifacts

When a step fails, the runner captures the current page state and writes a screenshot, so CI logs answer why the step failed instead of printing a bare timeout:

  ✗ [wait] the authenticated shell — wait for app-account-shell timed out after
30000ms: The event waited for never came — page: http://127.0.0.1:8082/auth/login
(Immosai) — visible: "Email address ⏎ Password ⏎ Sign in" (30.1s)
    │ url:      http://127.0.0.1:8082/auth/login
    │ title:    Immosai — Anmeldung
    │ content:  Email address  Password  ...  Invalid credentials.
      screenshot: artifacts/account__login-and-open-account__006-wait.png

Page state — URL, title, visible text, and any alert/error elements ([role="alert"], .error-message, snackbars, …) are appended to the step message and printed in full to stderr.
Screenshots — one PNG per failed step, written under --artifacts-dir (default artifacts/, env HARNESS_ARTIFACTS_DIR).
Fail fast — by default the first failed step ends the test and the remaining steps are reported as skipped (no LLM budget is burned asserting against a page that is already known broken). Set continue_on_failure = true in [config] or pass --continue-on-failure to keep executing every step.
LLM element targeting is verified — a response that is not a selector (:not(*), null, explanations, …) fails immediately with the raw LLM output; a selector that matches nothing triggers one retry with feedback.
LLM errors are specific — HTTP status, a truncated response-body snippet, and the attempt count are included, and deterministic client errors (401/403/404) fail fast instead of burning the retry budget.
Assertions always see the page — custom system+user_template definitions that omit the {content} placeholder automatically get the page URL/title/content appended, so the LLM never answers "I can't determine that without seeing the page".

Reporting: human- and machine-readable runs

All output flows through a single event stream. Every event (test/step started + finished, LLM call with duration/tokens/cost, budget warning) is rendered for humans and serialized for machines:

Console — level-filtered, ASCII-safe, colors only on a TTY (respects NO_COLOR). Default shows config, per-step results with durations, the run summary and the cost report. Use -q/-qq to hide step results (then warnings too), or -v/-vv to add LLM call details (endpoint, model, duration, tokens, cost) and step starts.
NDJSON log (--log-file run.jsonl) — one JSON object per event with a type discriminator and an epoch-ms ts field. Lossless apart from secret redaction: untruncated messages, ideal for CI artifact analysis:
```
jq '. | select(.type == "step_finished" and .status == "failed")' run.jsonl
jq '. | select(.type == "llm_call_finished") | {endpoint, ok, duration_ms, cost}'
```
JUnit XML (--junit report.xml) — one <testcase> per test with a <failure> per failed step, for Jenkins/GitLab/Azure/TeamCity.
Perfetto trace (--trace run.json) — test/step/LLM spans in Chrome Trace Event Format, viewable at https://ui.perfetto.dev.
GitHub Actions — in CI the reporter automatically emits ::error:: annotations for failed steps (with the screenshot as file=) and appends a run summary to GITHUB_STEP_SUMMARY.

Truncation (<truncated N chars>) is always boundary-safe (multi-byte UTF-8 can never panic it) and reports how much was cut; full text is preserved in the NDJSON log.

Secret redaction

Every sink is redacted through one chokepoint, so a leaked secret can never make it into a log or CI report:

API keys — llm_api_key and per-endpoint api_key
Static credentials — Entra auth.client_secret, AWS secret_access_key / session_token
Sensitive header values — values of authorization-family headers (matched case-insensitively: authorization, api-key, x-api-key, x-auth-token, token, cookie, …) in llm_headers and endpoint headers
Runtime-obtained tokens — token-command and header-command output, Entra client-credentials and managed-identity access tokens (registered the moment they are fetched, so the LlmCallFinished event that echoes them is redacted too)
Explicit extras — --redact <SECRET> (repeatable) or the HARNESS_REDACT env var (comma-separated), for secrets not present in the config (URL query tokens, scenario-embedded test data):
```
llm-browser-testkit run scenario.toml --redact 'abc123tokenxyz'
HARNESS_REDACT='abc123tokenxyz,xyz789tokenabc' llm-browser-testkit run scenario.toml
```

Secrets shorter than 6 characters are skipped when derived from config or runtime sources so short values (e.g. "dev") do not destroy log readability; explicit --redact values always apply. Replacement is exact-match and case-sensitive. Raw eprintln! sites that bypass the reporter (MCP/A2A server startup banners, the #[browser_test] run-report strings, the cost report) are not redacted.

Vision assertions (screenshots)

Text/DOM evaluation cannot see how the page renders — overlapping elements, clipped text, or a cookie banner covering the content are invisible to innerText. Mark an endpoint as vision-capable and attach a screenshot to an assert step to let the LLM evaluate the actual pixels:

[config]                      # optional: cap the screenshot resolution
screenshot_max_dimension = 1400

[config.endpoints.vision]     # MUST declare vision = true
type = "llm"
url = "https://api.openai.com"
model = "gpt-4o"
api_key = "sk-..."
vision = true                 # ← the flag
pricing = { input_per_1m_tokens = 2.50, output_per_1m_tokens = 10.00 }

[[definitions]]
name = "no_overlaps"
preset = "visual_no_overlaps"

[[test.steps]]
kind = "assert"
definition = "no_overlaps"
endpoint = "vision"
screenshot = true             # ← attach the viewport screenshot

How it works:

The viewport is captured as PNG, downscaled in Rust (Lanczos) so its longest edge is at most screenshot_max_dimension (default 1400), and re-encoded as quality-85 JPEG — no page JS, deterministic, and cheap on vision tokens.
The image is sent as an OpenAI-compatible image_url content part next to the text prompt (which still includes the page text for context).
Built-in presets: visual_no_issues, visual_no_overlaps, visual_text_visible (uses assert_text). Custom screenshot = true prompts work too.
A screenshot = true step that resolves to an endpoint without vision = true fails immediately with a clear configuration error.
Text-only workflows are untouched: without screenshot = true the request keeps the plain string content shape.

See examples/visual-overlays.toml + the bundled examples/visual-test-page.html fixture for a runnable demo that passes on a clean page and detects a cookie banner overlay. The prompts of the visual presets are intentionally strict ("only fail on clearly visible, user-impacting defects") — tune them per app if your overlay detection needs to be more or less sensitive.

DOM layout assertions (`layout_no_issues`)

Vision models see pixels but cost money per page × viewport. For cheap, deterministic layout coverage there is a DOM-only preset that never calls the LLM:

[[test.steps]]
kind = "assert"
preset = "layout_no_issues"   # no endpoint, no screenshot, no tokens

It evaluates a geometry scan in the page and fails with the detected issues:

page-overflow-x — the document is wider than the viewport (horizontal scrolling or a runaway element);
element-out-of-viewport — a visible element that scrolling cannot reveal: a fixed element outside the viewport, left/negative overflow, right-edge overflow beyond the scrollable content, or bottom overflow on a page that cannot scroll down. Below-the-fold content on a tall scrollable page is normal flow and is NOT reported;
text-clipped — content inside an overflow: hidden container is measurably larger than the box (cut-off text);
element-overlap — an interactive element's center point is covered by a different element that would intercept the click.

Intentional stacking (off-canvas drawers, dropdowns, badges, sticky headers) is excluded by position/relation filters. Elements whose class matches a prefix in [config] layout_ignore_classes are skipped by the fixed-element, text-clipped, and overlap checks — the default covers the Angular CDK screen-reader helpers (.cdk-visually-hidden, .cdk-describedby-message-container, .cdk-overlay-container), which are intentionally 1x1 / off-screen:

[config]
layout_ignore_classes = ["cdk-visually-hidden", "cdk-describedby-message-container", "cdk-overlay-container"]

Run it after every page load — it is free, so it is also the perfect companion for the viewport matrix below.

Viewport matrix (mobile / tablet / desktop)

[config.viewport_matrix] expands every test in a scenario into one variant per named viewport. Each variant overrides the browser viewport via CDP device-metrics emulation and gets a — <name> suffix on the test name; per-test budgets apply per variant.

[config.viewport_matrix]
viewports = [
  { name = "mobile",  width = 390,  height = 844 },
  { name = "tablet",  width = 768,  height = 1024 },
  { name = "desktop", width = 1280, height = 720 },
]

[[test]]
name = "Dashboard renders"
steps = [
    { kind = "navigate", url = "/dashboard", wait_after_ms = 2000 },
    { kind = "assert", preset = "layout_no_issues" },
]

The above runs "Dashboard renders — mobile", "— tablet" and "— desktop", each at its viewport, and the layout scan flags sticky overlays, off-screen text, or covered controls per size. Use it with screenshot = true + visual_no_issues on a vision endpoint for pixel-level checks on top.

Single-test overrides work too — any [[test]] may set viewport_width / viewport_height directly, which also switches the browser viewport via CDP for just that test:

[[test]]
name = "Narrow phone layout"
viewport_width = 320
viewport_height = 568

Assertion presets

Built-in presets you can use inline or from [[definitions]].

Preset	What it checks
`no_error_on_page`	No errors, stack traces, or broken UI on the page
`text_visible`	Specific text appears on the page (`assert_text`)
`element_exists`	A described UI element is present
`layout_no_issues`	DOM scan, no LLM: page overflow, elements out of viewport, clipped text, covered controls
`visual_no_issues`	Screenshot: no layout/rendering defects (overlaps, clipping, cut-off content, broken images, blank panels)
`visual_no_overlaps`	Screenshot: no elements covering other content or intercepting clicks
`visual_text_visible`	Screenshot: `assert_text` is fully visible and readable (not clipped or covered)

Custom assertions with prompt send any question to the LLM:

[[test.steps]]
kind = "assert"
prompt = "Does the page have a heading that says 'Example Domain'?"

Custom presets with system + user_template let you define reusable assertion logic with template variables {url}, {title}, {content}, {expected_text}, and {description}. Forgetting {content} is no longer a problem — the page context is appended automatically whenever the template does not reference it:

[[definitions]]
name = "text_matches"
system = "You are a QA tester."
user_template = "Does the page at {url} contain the text: {expected_text}?"
assert_text = "Welcome back"

CLI reference

llm-browser-testkit run <scenario.toml> [OPTIONS]

Flag	Default	Description
`--llm-url`	`$HARNESS_LLM_TEST_URL` or `http://localhost:8080`	OpenAI-compatible endpoint
`--llm-model`	`$HARNESS_LLM_TEST_MODEL` or `deepseek`	Model name
`--llm-api-key`	`$HARNESS_LLM_API_KEY`	API key (Bearer token)
`--llm-fallback-url`	`$HARNESS_LLM_FALLBACK_URL`	Fallback endpoint tried when the primary exhausts its attempts
`--llm-fallback-model`	`$HARNESS_LLM_FALLBACK_MODEL`	Fallback model name
`--llm-fallback-api-key`	`$HARNESS_LLM_FALLBACK_API_KEY`	Fallback API key
`--llm-header`	—	Custom header `Name:Value` (repeatable)
`--redact`	`$HARNESS_REDACT` (comma-separated)	Literal value redacted from all rows/sinks (repeatable)
`--model-param`	—	Provider param `key=value` (repeatable)
`--base-url`	`$HARNESS_BROWSER_BASE_URL` or `http://localhost:4200`	App under test
`--headless`	`true`	Run Chrome headlessly
`--timeout`	`60`	Seconds per action
`--viewport-width`	`1280`	Browser width
`--viewport-height`	`720`	Browser height
`--start-url`	`/dashboard`	First page to load
`--max-cost`	—	Global budget: max USD across all tests
`--max-tokens`	—	Global budget: max tokens across all tests
`--budget-enforcement`	`hard`	Budget mode: `hard` (abort) or `soft` (warn)
`--artifacts-dir`	`$HARNESS_ARTIFACTS_DIR` or `artifacts`	Directory for failure screenshots
`--continue-on-failure`	off	Keep running remaining steps after a step failure (default: fail fast)
`-q`, `--quiet`	off	Repeatable: hide step results (`-q`), then warnings too (`-qq`). Failures and the run summary always show
`-v`, `--verbose`	off	Repeatable: show LLM call details and step starts (`-v`), then everything (`-vv`)
`--log-file`	—	Write a machine-readable NDJSON event log (one JSON object per event, `type` + `ts` fields)
`--junit`	—	Write a JUnit XML report for CI systems
`--trace`	—	Write a Perfetto-format trace (test/step/LLM spans)
`--color`	`auto`	Console colors: `auto` (TTY + `NO_COLOR` aware), `always`, `never`

CLI flags override the scenario [config].

Retry + fallback behavior: every LLM call is retried up to HARNESS_LLM_CALL_ATTEMPTS times (default 3) on transient failures (network errors, HTTP 429/5xx, invalid JSON, and HTTP 200 with an empty body — the gateway warm-up signature). When an endpoint still fails, the fallback chain is tried: --llm-fallback-url/--llm-fallback-model/ --llm-fallback-api-key (or $HARNESS_LLM_FALLBACK_*) configure a single fallback endpoint for the implicit default endpoint. Pair a cheap primary with a more expensive, more powerful fallback — the fallback is only billed when the primary fails. Scenarios that declare [config.endpoints] use per-endpoint fallbacks = [...] instead (see below).

Endpoints

Define multiple named endpoints — LLM providers, MCP servers, and A2A agents — each with their own pricing, and route test steps to them automatically or explicitly.

[config.endpoints.default]
type = "llm"
url = "https://api.openai.com"
model = "gpt-4o-mini"
api_key = "sk-..."
pricing = { input_per_1m_tokens = 0.15, output_per_1m_tokens = 0.60 }
default_for = ["targeting", "assertion"]

[config.endpoints.vision]
type = "llm"
url = "https://api.openai.com"
model = "gpt-4o"
api_key = "sk-..."
pricing = { input_per_1m_tokens = 2.50, output_per_1m_tokens = 10.00 }
default_for = []

[config.endpoints.db_mcp]
type = "mcp"
command = "npx"
args = ["-y", "@modelcontextprotocol/server-postgres", "postgresql://localhost/mydb"]
pricing = { per_call = 0.001 }

[config.endpoints.audit_agent]
type = "a2a"
url = "http://localhost:9090"
pricing = { per_call = 0.01 }

[[test]]
name = "Dashboard with vision"

[[test.steps]]
kind = "navigate"
url = "/dashboard"

# Use the vision endpoint just for this assertion
[[test.steps]]
kind = "assert"
preset = "no_error_on_page"
endpoint = "vision"

Endpoint types:

llm — LLM chat API. Defaults to the OpenAI-compatible chat completions endpoint; provider = "azure" (Azure OpenAI) and provider = "bedrock" (AWS Bedrock, see below) are supported. Pricing is per-token (input_per_1m_tokens, output_per_1m_tokens).
mcp — Model Context Protocol server. Launched as a subprocess via command + args. Pricing is per_call.
a2a — Agent-to-Agent Protocol agent. Communicates via JSON-RPC over HTTP at the given url. Pricing is per_call.

Routing:

default_for lists which task types an endpoint serves automatically (targeting for element resolution, assertion for assertions) — this is the per-task model specification: give each task type its own endpoint (e.g. a cheap model for targeting, a stronger one for assertions) by splitting default_for across endpoints.
Add endpoint = "name" on any step or [[test]] group to override routing.

Retries + fallback chains:

max_attempts (per endpoint, default 3; global env override HARNESS_LLM_CALL_ATTEMPTS) — how often a single chat completion is retried on transient failures before the endpoint is considered failed.

fallbacks = ["other_endpoint", ...] (LLM endpoints only) — an ordered chain: when an endpoint exhausts its attempts, the next fallback is tried, and so on, until one answers. The answering endpoint is the one charged (per-usage cost reporting attributes the call correctly). Practical pattern: a cheap primary model with a more powerful, more expensive fallback that is only billed when the primary fails.

# Cheap by default; escalate to a stronger model when the gateway is
# down or returns garbage. Both endpoints serve both task types.
[config.endpoints.default]
type = "llm"
url = "$LLM_URL"            # home gateway / cheap model
model = "deepseek-v3"
default_for = ["targeting", "assertion"]
max_attempts = 5            # be patient with the local gateway
fallbacks = ["pro"]         # escalate only after 5 attempts

[config.endpoints.pro]
type = "llm"
url = "https://api.openai.com"
model = "gpt-4.1"
api_key = "sk-..."
pricing = { input_per_1m_tokens = 2.00, output_per_1m_tokens = 8.00 }
default_for = []

Chains are cycle-guarded and deduplicated; non-LLM endpoints in a fallbacks list are skipped. When all endpoints fail, the error message names every endpoint and its failure.

LLM providers: Azure & AWS Bedrock

Besides the default OpenAI-compatible API, LLM endpoints can target Azure OpenAI and AWS Bedrock.

Azure OpenAI

[config.endpoints.azure]
type = "llm"
provider = "azure"
url = "https://my-resource.openai.azure.com"   # resource endpoint, no path
deployment = "gpt-4o"                          # defaults to `model` when unset
api_version = "2024-10-21"                     # default when unset
api_key = "..."                                # sent as the `api-key` header
model = "gpt-4o"                               # unused by Azure; kept for pricing model names
pricing = { input_per_1m_tokens = 2.50, output_per_1m_tokens = 10.00 }
default_for = ["targeting", "assertion"]

Requests go to <url>/openai/deployments/<deployment>/chat/completions?api-version=<v>. With the default api-key auth mode the key is sent in the api-key header (Azure's classic convention). Use auth.api_key_header to send an API key in any custom header on any provider.

Instead of a static key you can authenticate with Entra ID — client credentials or a managed identity (see LLM authentication below).

AWS Bedrock

[config.endpoints.bedrock]
type = "llm"
provider = "bedrock"
model = "anthropic.claude-3-5-sonnet-20241022-v2:0"
region = "eu-central-1"          # optional: overrides the chain default
profile = "staging"              # optional: named profile from ~/.aws
pricing = { input_per_1m_tokens = 3.00, output_per_1m_tokens = 15.00 }
default_for = ["targeting", "assertion"]

# Optional: explicit credentials instead of the credential chain
# [config.endpoints.bedrock.aws]
# access_key_id = "AKIA..."
# secret_access_key = "..."
# session_token = "..."          # only for temporary credentials

Requests are signed with SigV4 and sent to https://bedrock-runtime.<region>.amazonaws.com/model/<model>/converse; the system prompt, temperature, maxTokens, and vision screenshots map to the Converse API (images become image content blocks). When no credentials are configured, the standard AWS credential chain is used — AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_PROFILE env vars, ~/.aws/config and ~/.aws/credentials, SSO, ECS and EC2 IMDS — exactly like the AWS CLI. The resolved credentials (and region) are cached per endpoint config for the process lifetime. Requires building with the aws cargo feature (cargo run --features aws ...); the Docker image and release binaries include it.

A2A agents

Call remote A2A agents in your test scenarios as steps, or use them inside assertion definitions for reusable agent-backed checks.

Agent step

[config.endpoints.audit_bot]
type = "a2a"
url = "http://localhost:9090"
pricing = { per_call = 0.01 }

[[test]]
name = "Audit trail check"
steps = [
    { kind = "navigate", url = "/admin/audit" },
    { kind = "agent", agent = "audit_bot", task = "Check if user 'admin' appears in the recent audit log" },
]

Agent-backed assertions

Define reusable agent assertions with task_template:

[[definitions]]
name = "audit_verify"
agent = "audit_bot"
task_template = "Verify that {expected_text} is true for the page at {url}"

[[test.steps]]
kind = "assert"
definition = "audit_verify"
assert_text = "the user can see the dashboard"

Template variables available: {url}, {title}, {content}, {expected_text}, {description}, {task}.

Run as an A2A agent

Enable the a2a-server feature to expose the framework as an A2A agent that other agents or orchestrators can call. The server listens on a port and accepts tasks/send JSON-RPC requests.

[config.a2a_server]
enabled = true
port = 3100

# Build and run with the a2a-server feature
cargo run --features a2a-server -- run scenario.toml --agent-port 3100

Or via CLI without modifying the TOML:

llm-browser-testkit run scenario.toml --agent-port 3100

Docker deployment

docker build -t llm-browser-testkit .
docker run --rm \
  -e HARNESS_LLM_TEST_URL=https://api.openai.com \
  -e HARNESS_LLM_TEST_MODEL=gpt-4o-mini \
  -e HARNESS_LLM_API_KEY=sk-... \
  llm-browser-testkit run scenario.toml --agent-port 3100 -p 3100:3100

A Dockerfile is included in the repository — it uses a multi-stage build with Alpine and Chromium. The image is built with --all-features, so all LLM providers (including Azure and AWS Bedrock) are available out of the box.

MCP tools

Call MCP server tools directly from test steps to query databases, read files, or invoke any tool an MCP server exposes.

[config.endpoints.db]
type = "mcp"
command = "npx"
args = ["-y", "@modelcontextprotocol/server-postgres", "postgresql://localhost/mydb"]
pricing = { per_call = 0.001 }

[[test]]
name = "Database smoke test"
steps = [
    { kind = "navigate", url = "/dashboard" },
    { kind = "mcp", server = "db", tool = "query", args = { sql = "SELECT count(*) FROM users" } },
    { kind = "assert", preset = "no_error_on_page" },
]

MCP servers are launched as subprocesses via the configured command and args. The framework handles the MCP initialize handshake, tool listing, and invocation automatically.

MCP server exposure

Enable the mcp-server feature to expose the framework as an MCP server so other tools can invoke it remotely.

[config.mcp_server]
enabled = true
port = 3000

cargo run --features mcp-server -- run scenario.toml

When enabled, other MCP clients can call tools like run_scenario and get_page_state on port 3000.

Cost tracking & budgets

Every LLM call, agent invocation, and MCP tool call is tracked. After the run completes, a cost report is printed with per-test and per-endpoint breakdowns.

Per-test default budget

[config.budgets.per_test_default]
max_cost = 1.0
max_tokens = 100_000
max_calls = 50
enforcement = "hard"

Per-test override

[[test]]
name = "Expensive test"
budget = { max_cost = 2.0, max_tokens = 200_000, enforcement = "soft" }

Global budget

[config.budgets.global]
max_cost = 5.0
max_tokens = 500_000
enforcement = "hard"

Enforcement

Mode	Behavior
`hard`	Abort the test or run immediately when budget is exceeded
`soft`	Print a warning but continue executing remaining steps

CLI budgets

llm-browser-testkit run scenario.toml --max-cost 10.0 --max-tokens 1000000 --budget-enforcement soft

Sample report output

-------------------------------
  COST REPORT
-------------------------------
  Test: "Homepage loads" — $0.0123 | 1,234 tokens | 4 calls
    endpoint.default:   4 calls,   1,234 tokens, $0.0123
  Test: "Dashboard smoke" — $0.0891 | 4,567 tokens | 6 calls
    endpoint.vision:    2 calls,   3,000 tokens, $0.0450
    endpoint.default:   3 calls,   1,567 tokens, $0.0441
    endpoint.audit_bot:   1 call,   0 tokens, $0.0000
-------------------------------
  GLOBAL SUMMARY
    Total cost:     $0.1014
    Total tokens:   5,801
    Total calls:    10
-------------------------------

How it works

Four pieces:

Chrome — launched via the Chrome DevTools Protocol (headless_chrome crate). It navigates, clicks, types, and extracts page content.
LLM — any OpenAI-compatible API. Used in two places:
- Element targeting: when a step says target = "the login button", the runner sends the page's interactive elements to the LLM and asks for a CSS selector.
- Assertions: the runner sends page content to the LLM with a QA prompt and expects PASS or FAIL: <reason>.
A2A + MCP — connect to remote agents via the Agent-to-Agent Protocol and to MCP servers for tool-calling. Both are first-class step kinds.
TOML scenarios — declarative test files. No code, no CSS selectors required. Just describe what you want in English.

TOML file  →  CLI runner  →  Chrome (CDP)  →  LLM API
                                        →  A2A agent
                                        →  MCP server

Use as a library

[dependencies]
llm-browser-testkit = { version = "0.1", features = ["macros", "mcp-server"] }

use llm_browser_testkit::runner::ScenarioRunner;
use llm_browser_testkit::scenario::Scenario;

let scenario: Scenario = toml::from_str(&contents)?;
let runner = ScenarioRunner::new(scenario.config.clone(), scenario.definitions);
let report = runner.run(&scenario.test)?;

println!("Passed: {}, Failed: {}", report.tests_passed, report.tests_failed);

// Access cost/usage data
let usage = runner.usage_tracker();
let global = usage.global_snapshot();
println!("Total cost: ${:.4}", global.total_cost);

// Print the cost report
llm_browser_testkit::reporting::print_report(
    &usage.per_test_snapshots(),
    &global,
);

Macros: `#[browser_test]` in `cargo test`

Enable the macros feature to write browser tests directly in your Rust test modules:

[dev-dependencies]
llm-browser-testkit = { version = "0.1", features = ["macros"] }

use llm_browser_testkit::browser_test;
use llm_browser_testkit::browser_test_inline;

// Run a TOML scenario file
browser_test!(homepage => "tests/homepage.toml");

// Inline small scenarios
browser_test_inline!(hello, r#"
[config]
base_url = "https://example.com"

[[test]]
name = "hello"

[[test.steps]]
kind = "navigate"
url = "/"

[[test.steps]]
kind = "assert"
preset = "no_error_on_page"
"#);

Tests auto-skip when no LLM endpoint or Chrome is available — safe to include in every CI run. They only execute with real PASS/FAIL when infrastructure is present.

LLM authentication

The runner supports API keys, Entra ID tokens, token commands, and custom (static or command-produced) headers — per endpoint.

API key (default api-key mode): sent as Authorization: Bearer <key> on OpenAI-compatible endpoints, as the api-key header on Azure. Set auth.api_key_header to use a different header name on any provider:

[config.endpoints.api]
type = "llm"
url = "https://api.openai.com"
api_key = "sk-..."
model = "gpt-4o"

[config.endpoints.api.auth]
mode = "api-key"                # default; can be omitted
api_key_header = "X-Api-Key"    # send the key here instead of Authorization

Custom headers — static headers and per-call header_commands (provider-agnostic; the command's first stdout line becomes the header value):

[config.endpoints.prod]
type = "llm"
url = "https://api.example.com"
model = "gpt-4o"
headers = { "X-Org-ID" = "acme" }            # static

[config.endpoints.prod.header_commands]
X-M2M-Token = "kubectl exec tokenizer -- token"   # dynamic, per call

Token commands (mode = "token-command"): run any program, its stdout (first line) becomes the bearer token. Works with any CLI that prints a token — Azure CLI, Vault, ...:

[config.endpoints.azure]
type = "llm"
provider = "azure"
url = "https://my-resource.openai.azure.com"
model = "gpt-4o"

[config.endpoints.azure.auth]
mode = "token-command"
token_command = "az account get-access-token --resource https://cognitiveservices.azure.com --query accessToken -o tsv"
cache_ttl_secs = 240            # reuse the token this long (default 300)

Entra ID client credentials (mode = "entra-client-credentials"): the OAuth 2.0 client-credentials grant against https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token; the token is cached until its server-issued expiry, then refreshed automatically:

[config.endpoints.azure.auth]
mode = "entra-client-credentials"
tenant_id = "<tenant-or-uuid>"
client_id = "<app-registration-client-id>"
client_secret = "<client-secret>"
scope = "https://cognitiveservices.azure.com/.default"   # default

Entra ID managed identity (mode = "entra-managed-identity"): fetches a token from the Azure IMDS endpoint — zero credentials in the config; works on Azure VMs, App Service, and ACI with a system-assigned identity:

[config.endpoints.azure.auth]
mode = "entra-managed-identity"

Token acquisition is cached process-wide per auth configuration, so a test run authenticates once instead of on every LLM call.

Via CLI / env, the runner still supports plain keys and headers:

llm-browser-testkit run tests.toml \
  --llm-api-key sk-... \
  --llm-header "X-Org-ID:acme" \
  --llm-header "X-Project:qa"

export HARNESS_LLM_API_KEY=sk-...
export HARNESS_LLM_HEADERS='{"X-Org-ID":"acme","X-Project":"qa"}'

License

Apache-2.0 OR MIT

llm-browser-testkit 0.10.0

llm-browser-testkit

Contents

Quick start

Write your first test

Step reference

Failure diagnostics & artifacts

Reporting: human- and machine-readable runs

Secret redaction

Vision assertions (screenshots)

DOM layout assertions (`layout_no_issues`)

Viewport matrix (mobile / tablet / desktop)

Assertion presets

CLI reference

Endpoints

LLM providers: Azure & AWS Bedrock

Azure OpenAI

AWS Bedrock

A2A agents

Agent step

Agent-backed assertions

Run as an A2A agent

Docker deployment

MCP tools

MCP server exposure

Cost tracking & budgets

Per-test default budget

Per-test override

Global budget

Enforcement

CLI budgets

Sample report output

How it works

Use as a library

Macros: `#[browser_test]` in `cargo test`

LLM authentication

License

llm-browser-testkit 0.10.0

llm-browser-testkit

Contents

Quick start

Write your first test

Step reference

Failure diagnostics & artifacts

Reporting: human- and machine-readable runs

Secret redaction

Vision assertions (screenshots)

DOM layout assertions (layout_no_issues)

Viewport matrix (mobile / tablet / desktop)

Assertion presets

CLI reference

Endpoints

LLM providers: Azure & AWS Bedrock

Azure OpenAI

AWS Bedrock

A2A agents

Agent step

Agent-backed assertions

Run as an A2A agent

Docker deployment

MCP tools

MCP server exposure

Cost tracking & budgets

Per-test default budget

Per-test override

Global budget

Enforcement

CLI budgets

Sample report output

How it works

Use as a library

Macros: #[browser_test] in cargo test

LLM authentication

License

DOM layout assertions (`layout_no_issues`)

Macros: `#[browser_test]` in `cargo test`