llm-browser-testkit

Describe browser tests in plain English. The LLM figures out which elements to click and whether the page looks right — plus A2A agents, MCP tool-calling, cost tracking, and budgets.

llm-browser-testkit run smoke.toml

Contents

• Quick start
• Write your first test
• Step reference
• Assertion presets
• CLI reference
• Endpoints
• A2A agents
• Run as an A2A agent
• MCP tools
• MCP server
• Cost tracking & budgets
• How it works
• Use as a library
• LLM authentication
• License

Quick start

# Install
cargo install llm-browser-testkit

# Set your LLM credentials (OpenAI-compatible API)
export HARNESS_LLM_TEST_URL=https://api.openai.com
export HARNESS_LLM_TEST_MODEL=gpt-4o-mini
export HARNESS_LLM_API_KEY=sk-...

# Run the built-in example (tests example.com — no account needed)
llm-browser-testkit run examples/smoke.toml

Write your first test

# hello.toml
[config]
base_url = "https://example.com"
timeout_secs = 30
start_url = "/"

[[definitions]]
name = "no_errors"
preset = "no_error_on_page"

[[test]]
name = "Homepage loads"

[[test.steps]]
kind = "navigate"
url = "/"

[[test.steps]]
kind = "assert"
definition = "no_errors"

Run it:

llm-browser-testkit run hello.toml

Step reference

Every step has a kind. Required fields depend on the kind.

kind	What it does	Required	Optional
navigate	Open a URL	url	wait_after_ms
click	Click an element	target	selector, wait_after_ms, endpoint, idempotent
type	Type into a field	target, text	selector, wait_after_ms, endpoint, idempotent
wait	Wait for an element and/or visible text	target	selector, text, timeout_ms, endpoint, idempotent
assert	Check the page	one of definition, preset, or prompt	assert_text, endpoint
screenshot	Save a .png	—	path
agent	Call an A2A agent	agent, task	definition
mcp	Call an MCP tool	server, tool	args

target is natural language ("the submit button", "the search input"). The LLM looks at the page DOM and picks the right CSS selector at runtime. Skip the LLM with an explicit selector.

idempotent — an optional flag on click, type and wait steps. When the step's target is absent, the step is reported skipped instead of failed: the action was already done or not applicable. This is the generic building block for flows that repeat in one browser session — e.g. logging in on every viewport-matrix variant of the same test:

[[test.steps]]
kind = "navigate"
url = "/auth/login"

# Already authenticated? The form is gone, so these steps skip
# instead of failing.
[[test.steps]]
kind = "type"
selector = "#email"
target = "the email input"
text = "admin@example.com"
idempotent = true

[[test.steps]]
kind = "type"
selector = "#password"
target = "the password input"
text = "correct horse battery staple"
idempotent = true

[[test.steps]]
kind = "wait"
selector = "input[name="cf-turnstile-response"][value]:not([value=""])"
target = "the bot-protection token"
timeout_ms = 15000
idempotent = true

[[test.steps]]
kind = "click"
selector = "button.btn--landing.btn--primary"
target = "the sign-in button"
idempotent = true

# The final check stays strict: a real login attempt that never
# reaches the authenticated shell still fails the test.
[[test.steps]]
kind = "wait"
selector = "app-account-shell"
target = "the authenticated shell"
timeout_ms = 30000

Semantics:

• idempotent click / type: probe for up to 5s; absent target → skipped.
• idempotent wait: run the wait as normal; a timeout → skipped instead of failed.
• Skipped steps do not fail the test and do not trigger fail-fast.
• Keep the final verification step strict (no idempotent) so real failures in the middle of an idempotent flow still surface.

endpoint routes this step to a specific endpoint. Use it to send element targeting to one model and assertions to another.

wait with text waits until the page's visible text contains a substring — no selector or LLM needed:

[[test.steps]]
kind = "wait"
target = "the success message"
text = "Welcome back"
timeout_ms = 5000

Set both selector and text to require both conditions. The combined wait shares one timeout_ms budget.

Failure diagnostics & artifacts

When a step fails, the runner captures the current page state and writes a screenshot, so CI logs answer why the step failed instead of printing a bare timeout:

❌ [wait] the authenticated shell — wait for app-account-shell timed out after
30000ms: The event waited for never came — page: http://127.0.0.1:8082/auth/login
(Immosai) — visible: "Email address ⏎ Password ⏎ Sign in"
    │ url:      http://127.0.0.1:8082/auth/login
    │ title:    Immosai — Anmeldung
    │ content:  Email address  Password  ...  Invalid credentials.
    📸 screenshot: artifacts/account__login-and-open-account__006-wait.png

• Page state — URL, title, visible text, and any alert/error elements ([role="alert"], .error-message, snackbars, …) are appended to the step message and printed in full to stderr.
• Screenshots — one PNG per failed step, written under --artifacts-dir (default artifacts/, env HARNESS_ARTIFACTS_DIR).
• Fail fast — by default the first failed step ends the test and the remaining steps are reported as skipped (no LLM budget is burned asserting against a page that is already known broken). Set continue_on_failure = true in [config] or pass --continue-on-failure to keep executing every step.
• LLM element targeting is verified — a response that is not a selector (:not(*), null, explanations, …) fails immediately with the raw LLM output; a selector that matches nothing triggers one retry with feedback.
• LLM errors are specific — HTTP status, a truncated response-body snippet, and the attempt count are included, and deterministic client errors (401/403/404) fail fast instead of burning the retry budget.
• Assertions always see the page — custom system+user_template definitions that omit the {content} placeholder automatically get the page URL/title/content appended, so the LLM never answers "I can't determine that without seeing the page".

Vision assertions (screenshots)

Text/DOM evaluation cannot see how the page renders — overlapping elements, clipped text, or a cookie banner covering the content are invisible to innerText. Mark an endpoint as vision-capable and attach a screenshot to an assert step to let the LLM evaluate the actual pixels:

[config]                      # optional: cap the screenshot resolution
screenshot_max_dimension = 1400

[config.endpoints.vision]     # MUST declare vision = true
type = "llm"
url = "https://api.openai.com"
model = "gpt-4o"
api_key = "sk-..."
vision = true                 # ← the flag
pricing = { input_per_1m_tokens = 2.50, output_per_1m_tokens = 10.00 }

[[definitions]]
name = "no_overlaps"
preset = "visual_no_overlaps"

[[test.steps]]
kind = "assert"
definition = "no_overlaps"
endpoint = "vision"
screenshot = true             # ← attach the viewport screenshot

How it works:

• The viewport is captured as PNG, downscaled in Rust (Lanczos) so its longest edge is at most screenshot_max_dimension (default 1400), and re-encoded as quality-85 JPEG — no page JS, deterministic, and cheap on vision tokens.
• The image is sent as an OpenAI-compatible image_url content part next to the text prompt (which still includes the page text for context).
• Built-in presets: visual_no_issues, visual_no_overlaps, visual_text_visible (uses assert_text). Custom screenshot = true prompts work too.
• A screenshot = true step that resolves to an endpoint without vision = true fails immediately with a clear configuration error.
• Text-only workflows are untouched: without screenshot = true the request keeps the plain string content shape.

See examples/visual-overlays.toml + the bundled examples/visual-test-page.html fixture for a runnable demo that passes on a clean page and detects a cookie banner overlay. The prompts of the visual presets are intentionally strict ("only fail on clearly visible, user-impacting defects") — tune them per app if your overlay detection needs to be more or less sensitive.

DOM layout assertions (layout_no_issues)

Vision models see pixels but cost money per page × viewport. For cheap, deterministic layout coverage there is a DOM-only preset that never calls the LLM:

[[test.steps]]
kind = "assert"
preset = "layout_no_issues"   # no endpoint, no screenshot, no tokens

It evaluates a geometry scan in the page and fails with the detected issues:

• page-overflow-x — the document is wider than the viewport (horizontal scrolling or a runaway element);
• element-out-of-viewport — a visible element that scrolling cannot reveal: a fixed element outside the viewport, left/negative overflow, right-edge overflow beyond the scrollable content, or bottom overflow on a page that cannot scroll down. Below-the-fold content on a tall scrollable page is normal flow and is NOT reported;
• text-clipped — content inside an overflow: hidden container is measurably larger than the box (cut-off text);
• element-overlap — an interactive element's center point is covered by a different element that would intercept the click.

Intentional stacking (off-canvas drawers, dropdowns, badges, sticky headers) is excluded by position/relation filters. Elements whose class matches a prefix in [config] layout_ignore_classes are skipped by the fixed-element, text-clipped, and overlap checks — the default covers the Angular CDK screen-reader helpers (.cdk-visually-hidden, .cdk-describedby-message-container, .cdk-overlay-container), which are intentionally 1x1 / off-screen:

[config]
layout_ignore_classes = ["cdk-visually-hidden", "cdk-describedby-message-container", "cdk-overlay-container"]

Run it after every page load — it is free, so it is also the perfect companion for the viewport matrix below.

Viewport matrix (mobile / tablet / desktop)

[config.viewport_matrix] expands every test in a scenario into one variant per named viewport. Each variant overrides the browser viewport via CDP device-metrics emulation and gets a — <name> suffix on the test name; per-test budgets apply per variant.

[config.viewport_matrix]
viewports = [
  { name = "mobile",  width = 390,  height = 844 },
  { name = "tablet",  width = 768,  height = 1024 },
  { name = "desktop", width = 1280, height = 720 },
]

[[test]]
name = "Dashboard renders"
steps = [
    { kind = "navigate", url = "/dashboard", wait_after_ms = 2000 },
    { kind = "assert", preset = "layout_no_issues" },
]

The above runs "Dashboard renders — mobile", "— tablet" and "— desktop", each at its viewport, and the layout scan flags sticky overlays, off-screen text, or covered controls per size. Use it with screenshot = true + visual_no_issues on a vision endpoint for pixel-level checks on top.

Single-test overrides work too — any [[test]] may set viewport_width / viewport_height directly, which also switches the browser viewport via CDP for just that test:

[[test]]
name = "Narrow phone layout"
viewport_width = 320
viewport_height = 568

Assertion presets

Built-in presets you can use inline or from [[definitions]].

Preset	What it checks
no_error_on_page	No errors, stack traces, or broken UI on the page
text_visible	Specific text appears on the page (assert_text)
element_exists	A described UI element is present
layout_no_issues	DOM scan, no LLM: page overflow, elements out of viewport, clipped text, covered controls
visual_no_issues	Screenshot: no layout/rendering defects (overlaps, clipping, cut-off content, broken images, blank panels)
visual_no_overlaps	Screenshot: no elements covering other content or intercepting clicks
visual_text_visible	Screenshot: assert_text is fully visible and readable (not clipped or covered)

Custom assertions with prompt send any question to the LLM:

[[test.steps]]
kind = "assert"
prompt = "Does the page have a heading that says 'Example Domain'?"

Custom presets with system + user_template let you define reusable assertion logic with template variables {url}, {title}, {content}, {expected_text}, and {description}. Forgetting {content} is no longer a problem — the page context is appended automatically whenever the template does not reference it:

[[definitions]]
name = "text_matches"
system = "You are a QA tester."
user_template = "Does the page at {url} contain the text: {expected_text}?"
assert_text = "Welcome back"

CLI reference

llm-browser-testkit run <scenario.toml> [OPTIONS]

Flag	Default	Description
--llm-url	$HARNESS_LLM_TEST_URL or http://localhost:8080	OpenAI-compatible endpoint
--llm-model	$HARNESS_LLM_TEST_MODEL or deepseek	Model name
--llm-api-key	$HARNESS_LLM_API_KEY	API key (Bearer token)
--llm-fallback-url	$HARNESS_LLM_FALLBACK_URL	Fallback endpoint tried when the primary exhausts its attempts
--llm-fallback-model	$HARNESS_LLM_FALLBACK_MODEL	Fallback model name
--llm-fallback-api-key	$HARNESS_LLM_FALLBACK_API_KEY	Fallback API key
--llm-header	—	Custom header Name:Value (repeatable)
--model-param	—	Provider param key=value (repeatable)
--base-url	$HARNESS_BROWSER_BASE_URL or http://localhost:4200	App under test
--headless	true	Run Chrome headlessly
--timeout	60	Seconds per action
--viewport-width	1280	Browser width
--viewport-height	720	Browser height
--start-url	/dashboard	First page to load
--max-cost	—	Global budget: max USD across all tests
--max-tokens	—	Global budget: max tokens across all tests
--budget-enforcement	hard	Budget mode: hard (abort) or soft (warn)
--artifacts-dir	$HARNESS_ARTIFACTS_DIR or artifacts	Directory for failure screenshots
--continue-on-failure	off	Keep running remaining steps after a step failure (default: fail fast)

CLI flags override the scenario [config].

Retry + fallback behavior: every LLM call is retried up to HARNESS_LLM_CALL_ATTEMPTS times (default 3) on transient failures (network errors, HTTP 429/5xx, invalid JSON, and HTTP 200 with an empty body — the gateway warm-up signature). When an endpoint still fails, the fallback chain is tried: --llm-fallback-url/--llm-fallback-model/ --llm-fallback-api-key (or $HARNESS_LLM_FALLBACK_*) configure a single fallback endpoint for the implicit default endpoint. Pair a cheap primary with a more expensive, more powerful fallback — the fallback is only billed when the primary fails. Scenarios that declare [config.endpoints] use per-endpoint fallbacks = [...] instead (see below).

Endpoints

Define multiple named endpoints — LLM providers, MCP servers, and A2A agents — each with their own pricing, and route test steps to them automatically or explicitly.

[config.endpoints.default]
type = "llm"
url = "https://api.openai.com"
model = "gpt-4o-mini"
api_key = "sk-..."
pricing = { input_per_1m_tokens = 0.15, output_per_1m_tokens = 0.60 }
default_for = ["targeting", "assertion"]

[config.endpoints.vision]
type = "llm"
url = "https://api.openai.com"
model = "gpt-4o"
api_key = "sk-..."
pricing = { input_per_1m_tokens = 2.50, output_per_1m_tokens = 10.00 }
default_for = []

[config.endpoints.db_mcp]
type = "mcp"
command = "npx"
args = ["-y", "@modelcontextprotocol/server-postgres", "postgresql://localhost/mydb"]
pricing = { per_call = 0.001 }

[config.endpoints.audit_agent]
type = "a2a"
url = "http://localhost:9090"
pricing = { per_call = 0.01 }

[[test]]
name = "Dashboard with vision"

[[test.steps]]
kind = "navigate"
url = "/dashboard"

# Use the vision endpoint just for this assertion
[[test.steps]]
kind = "assert"
preset = "no_error_on_page"
endpoint = "vision"

Endpoint types:

• llm — OpenAI-compatible chat completions API. Pricing is per-token (input_per_1m_tokens, output_per_1m_tokens).
• mcp — Model Context Protocol server. Launched as a subprocess via command + args. Pricing is per_call.
• a2a — Agent-to-Agent Protocol agent. Communicates via JSON-RPC over HTTP at the given url. Pricing is per_call.

Routing:

• default_for lists which task types an endpoint serves automatically (targeting for element resolution, assertion for assertions) — this is the per-task model specification: give each task type its own endpoint (e.g. a cheap model for targeting, a stronger one for assertions) by splitting default_for across endpoints.
• Add endpoint = "name" on any step or [[test]] group to override routing.

Retries + fallback chains:

• max_attempts (per endpoint, default 3; global env override HARNESS_LLM_CALL_ATTEMPTS) — how often a single chat completion is retried on transient failures before the endpoint is considered failed.

• fallbacks = ["other_endpoint", ...] (LLM endpoints only) — an ordered chain: when an endpoint exhausts its attempts, the next fallback is tried, and so on, until one answers. The answering endpoint is the one charged (per-usage cost reporting attributes the call correctly). Practical pattern: a cheap primary model with a more powerful, more expensive fallback that is only billed when the primary fails.

  # Cheap by default; escalate to a stronger model when the gateway is
  # down or returns garbage. Both endpoints serve both task types.
  [config.endpoints.default]
  type = "llm"
  url = "$LLM_URL"            # home gateway / cheap model
  model = "deepseek-v3"
  default_for = ["targeting", "assertion"]
  max_attempts = 5            # be patient with the local gateway
  fallbacks = ["pro"]         # escalate only after 5 attempts
  
  [config.endpoints.pro]
  type = "llm"
  url = "https://api.openai.com"
  model = "gpt-4.1"
  api_key = "sk-..."
  pricing = { input_per_1m_tokens = 2.00, output_per_1m_tokens = 8.00 }
  default_for = []
  

  Chains are cycle-guarded and deduplicated; non-LLM endpoints in a fallbacks list are skipped. When all endpoints fail, the error message names every endpoint and its failure.

A2A agents

Call remote A2A agents in your test scenarios as steps, or use them inside assertion definitions for reusable agent-backed checks.

Agent step

[config.endpoints.audit_bot]
type = "a2a"
url = "http://localhost:9090"
pricing = { per_call = 0.01 }

[[test]]
name = "Audit trail check"
steps = [
    { kind = "navigate", url = "/admin/audit" },
    { kind = "agent", agent = "audit_bot", task = "Check if user 'admin' appears in the recent audit log" },
]

Agent-backed assertions

Define reusable agent assertions with task_template:

[[definitions]]
name = "audit_verify"
agent = "audit_bot"
task_template = "Verify that {expected_text} is true for the page at {url}"

[[test.steps]]
kind = "assert"
definition = "audit_verify"
assert_text = "the user can see the dashboard"

Template variables available: {url}, {title}, {content}, {expected_text}, {description}, {task}.

Run as an A2A agent

Enable the a2a-server feature to expose the framework as an A2A agent that other agents or orchestrators can call. The server listens on a port and accepts tasks/send JSON-RPC requests.

[config.a2a_server]
enabled = true
port = 3100

# Build and run with the a2a-server feature
cargo run --features a2a-server -- run scenario.toml --agent-port 3100

Or via CLI without modifying the TOML:

llm-browser-testkit run scenario.toml --agent-port 3100

Docker deployment

docker build -t llm-browser-testkit .
docker run --rm \
  -e HARNESS_LLM_TEST_URL=https://api.openai.com \
  -e HARNESS_LLM_TEST_MODEL=gpt-4o-mini \
  -e HARNESS_LLM_API_KEY=sk-... \
  llm-browser-testkit run scenario.toml --agent-port 3100 -p 3100:3100

A Dockerfile is included in the repository — it uses a multi-stage build with Alpine and Chromium.

MCP tools

Call MCP server tools directly from test steps to query databases, read files, or invoke any tool an MCP server exposes.

[config.endpoints.db]
type = "mcp"
command = "npx"
args = ["-y", "@modelcontextprotocol/server-postgres", "postgresql://localhost/mydb"]
pricing = { per_call = 0.001 }

[[test]]
name = "Database smoke test"
steps = [
    { kind = "navigate", url = "/dashboard" },
    { kind = "mcp", server = "db", tool = "query", args = { sql = "SELECT count(*) FROM users" } },
    { kind = "assert", preset = "no_error_on_page" },
]

MCP servers are launched as subprocesses via the configured command and args. The framework handles the MCP initialize handshake, tool listing, and invocation automatically.

MCP server exposure

Enable the mcp-server feature to expose the framework as an MCP server so other tools can invoke it remotely.

[config.mcp_server]
enabled = true
port = 3000

cargo run --features mcp-server -- run scenario.toml

When enabled, other MCP clients can call tools like run_scenario and get_page_state on port 3000.

Cost tracking & budgets

Every LLM call, agent invocation, and MCP tool call is tracked. After the run completes, a cost report is printed with per-test and per-endpoint breakdowns.

Per-test default budget

[config.budgets.per_test_default]
max_cost = 1.0
max_tokens = 100_000
max_calls = 50
enforcement = "hard"

Per-test override

[[test]]
name = "Expensive test"
budget = { max_cost = 2.0, max_tokens = 200_000, enforcement = "soft" }

Global budget

[config.budgets.global]
max_cost = 5.0
max_tokens = 500_000
enforcement = "hard"

Enforcement

Mode	Behavior
hard	Abort the test or run immediately when budget is exceeded
soft	Print a warning but continue executing remaining steps

CLI budgets

llm-browser-testkit run scenario.toml --max-cost 10.0 --max-tokens 1000000 --budget-enforcement soft

Sample report output

═══════════════════════════════════════════════
  COST REPORT
═══════════════════════════════════════════════
  Test: "Homepage loads" — $0.0123 | 1,234 tokens | 4 calls
    endpoint.default:   4 calls,   1,234 tokens, $0.0123
  Test: "Dashboard smoke" — $0.0891 | 4,567 tokens | 6 calls
    endpoint.vision:    2 calls,   3,000 tokens, $0.0450
    endpoint.default:   3 calls,   1,567 tokens, $0.0441
    endpoint.audit_bot:   1 call,   0 tokens, $0.0000
───────────────────────────────────────────────
  GLOBAL SUMMARY
    Total cost:     $0.1014
    Total tokens:   5,801
    Total calls:    10
═══════════════════════════════════════════════

How it works

Four pieces:

1. Chrome — launched via the Chrome DevTools Protocol (headless_chrome crate). It navigates, clicks, types, and extracts page content.

2. LLM — any OpenAI-compatible API. Used in two places:

   • Element targeting: when a step says target = "the login button", the runner sends the page's interactive elements to the LLM and asks for a CSS selector.
   • Assertions: the runner sends page content to the LLM with a QA prompt and expects PASS or FAIL: <reason>.

3. A2A + MCP — connect to remote agents via the Agent-to-Agent Protocol and to MCP servers for tool-calling. Both are first-class step kinds.

4. TOML scenarios — declarative test files. No code, no CSS selectors required. Just describe what you want in English.

TOML file  →  CLI runner  →  Chrome (CDP)  →  LLM API
                                        →  A2A agent
                                        →  MCP server

Use as a library

[dependencies]
llm-browser-testkit = { version = "0.1", features = ["macros", "mcp-server"] }

use llm_browser_testkit::runner::ScenarioRunner;
use llm_browser_testkit::scenario::Scenario;

let scenario: Scenario = toml::from_str(&contents)?;
let runner = ScenarioRunner::new(scenario.config.clone(), scenario.definitions);
let report = runner.run(&scenario.test)?;

println!("Passed: {}, Failed: {}", report.tests_passed, report.tests_failed);

// Access cost/usage data
let usage = runner.usage_tracker();
let global = usage.global_snapshot();
println!("Total cost: ${:.4}", global.total_cost);

// Print the cost report
llm_browser_testkit::reporting::print_report(
    &usage.per_test_snapshots(),
    &global,
);

Macros: #[browser_test] in cargo test

Enable the macros feature to write browser tests directly in your Rust test modules:

[dev-dependencies]
llm-browser-testkit = { version = "0.1", features = ["macros"] }

use llm_browser_testkit::browser_test;
use llm_browser_testkit::browser_test_inline;

// Run a TOML scenario file
browser_test!(homepage => "tests/homepage.toml");

// Inline small scenarios
browser_test_inline!(hello, r#"
[config]
base_url = "https://example.com"

[[test]]
name = "hello"

[[test.steps]]
kind = "navigate"
url = "/"

[[test.steps]]
kind = "assert"
preset = "no_error_on_page"
"#);

Tests auto-skip when no LLM endpoint or Chrome is available — safe to include in every CI run. They only execute with real PASS/FAIL when infrastructure is present.

LLM authentication

The runner supports API keys and custom headers for SSO or alternative auth:

[config]
llm_api_key = "sk-..."
llm_headers = { "X-Org-ID" = "acme", "X-Project" = "qa" }

Endpoints can also carry their own credentials:

[config.endpoints.production]
type = "llm"
url = "https://api.openai.com"
api_key = "sk-prod-..."
model = "gpt-4o"

Via CLI:

llm-browser-testkit run tests.toml \
  --llm-api-key sk-... \
  --llm-header "X-Org-ID:acme" \
  --llm-header "X-Project:qa"

Via env:

export HARNESS_LLM_API_KEY=sk-...
export HARNESS_LLM_HEADERS='{"X-Org-ID":"acme","X-Project":"qa"}'

License

Apache-2.0 OR MIT