llm-browser-testkit 0.18.1

# llm-browser-testkit

Describe browser tests in plain English. The LLM figures out which elements to
click and whether the page looks right — plus A2A agents, MCP tool-calling, cost
tracking, and budgets.

```
llm-browser-testkit run smoke.toml
```

## Contents

- [Quick start](#quick-start)
- [Write your first test](#write-your-first-test)
- [Step reference](#step-reference)
- [Assertion presets](#assertion-presets)
- [CLI reference](#cli-reference)
- [Endpoints](#endpoints)
- [A2A agents](#a2a-agents)
- [Run as an A2A agent](#run-as-an-a2a-agent)
- [MCP tools](#mcp-tools)
- [MCP server](#mcp-server-exposure)
- [Cost tracking & budgets](#cost-tracking--budgets)
- [Parallel runs](#parallel-runs)
- [How it works](#how-it-works)
- [Use as a library](#use-as-a-library)
- [LLM authentication](#llm-authentication)
- [License](#license)

## Quick start

Everything you need for a first green run — copy-paste, no prior setup:

```bash
# 1. Install the CLI
cargo install llm-browser-testkit

# 2. Point it at any OpenAI-compatible API
export HARNESS_LLM_TEST_URL=https://api.openai.com
export HARNESS_LLM_TEST_MODEL=gpt-4o-mini
export HARNESS_LLM_API_KEY=sk-...

# 3. Describe one test in a tiny TOML file (example.com — no account needed)
cat > hello.toml <<'EOF'
[config]
base_url = "https://example.com"
start_url = "/"

[[test]]
name = "Homepage loads"

[[test.steps]]
kind = "navigate"
url = "/"

[[test.steps]]
kind = "assert"
preset = "no_error_on_page"
EOF

# 4. Run it — Chrome runs headless, the LLM checks the page
llm-browser-testkit run hello.toml
```

Example output:

```
Test: Homepage loads — passed (6.2s, $0.0005, 138 tokens (100 in / 38 out, 0 cached, 0 cache write), 1 calls, 2+0+0 steps) | models: deepseek
run passed: tests 1 passed, 0 failed | steps 2 passed, 0 failed, 0 skipped | $0.0005 | 138 tokens (100 in / 38 out, 0 cached, 0 cache write) | 1 calls | models: deepseek
```

## Write your first test

The quick-start `hello.toml` is the smallest useful scenario. Its three
blocks:

- `[config]` — `base_url` is the app under test; `start_url` is where Chrome
  loads first.
- `[[test]]` — one named test, built from `[[test.steps]]` that run top to
  bottom.
- Steps — every step has a `kind`: `navigate` opens a URL relative to
  `base_url`; `assert` sends the page to the LLM and expects `PASS`/`FAIL`.
  `preset` picks a built-in check (`no_error_on_page` = no errors, stack
  traces, or broken UI).

Reusable checks go into `[[definitions]]` — name a preset or prompt once and
reference it from any assertion:

```toml
[[definitions]]
name = "no_errors"
preset = "no_error_on_page"

[[definitions]]
name = "example_domain_visible"
preset = "text_visible"
assert_text = "Example Domain"

[[test]]
name = "Homepage loads"

[[test.steps]]
kind = "navigate"
url = "/"

[[test.steps]]
kind = "assert"
definition = "no_errors"

[[test.steps]]
kind = "assert"
definition = "example_domain_visible"
```

Run it:

```bash
llm-browser-testkit run hello.toml
```

## Step reference

Every step has a `kind`. Required fields depend on the kind.

| `kind` | What it does | Required | Optional |
|--------|-------------|----------|----------|
| `navigate` | Open a URL | `url` | `wait_after_ms` |
| `click` | Click an element | `target` | `selector`, `wait_after_ms`, `endpoint`, `idempotent` |
| `type` | Type into a field | `target`, `text` | `selector`, `wait_after_ms`, `endpoint`, `idempotent` |
| `wait` | Wait for an element and/or visible text | `target` | `selector`, `text`, `timeout_ms`, `endpoint`, `idempotent` |
| `assert` | Check the page | one of `definition`, `preset`, or `prompt` | `assert_text`, `endpoint` |
| `screenshot` | Save a .png | — | `path` |
| `agent` | Call an A2A agent | `agent`, `task` | `definition` |
| `mcp` | Call an MCP tool | `server`, `tool` | `args` |

**`target`** is natural language ("the submit button", "the search input"). The
LLM looks at the page DOM and picks the right CSS selector at runtime. Skip the
LLM with an explicit `selector`.

**`idempotent`** — an optional flag on `click`, `type` and `wait` steps.
When the step's target is absent, the step is reported **skipped** instead
of failed: the action was already done or not applicable. This is the
generic building block for flows that repeat in one browser session —
e.g. logging in on every viewport-matrix variant of the same test:

```toml
[[test.steps]]
kind = "navigate"
url = "/auth/login"

# Already authenticated? The form is gone, so these steps skip
# instead of failing.
[[test.steps]]
kind = "type"
selector = "#email"
target = "the email input"
text = "admin@example.com"
idempotent = true

[[test.steps]]
kind = "type"
selector = "#password"
target = "the password input"
text = "correct horse battery staple"
idempotent = true

[[test.steps]]
kind = "wait"
selector = "input[name="cf-turnstile-response"][value]:not([value=""])"
target = "the bot-protection token"
timeout_ms = 15000
idempotent = true

[[test.steps]]
kind = "click"
selector = "button.btn--landing.btn--primary"
target = "the sign-in button"
idempotent = true

# The final check stays strict: a real login attempt that never
# reaches the authenticated shell still fails the test.
[[test.steps]]
kind = "wait"
selector = "app-account-shell"
target = "the authenticated shell"
timeout_ms = 30000
```

Semantics:
- `idempotent` `click` / `type`: probe for up to 5s; absent target → skipped.
- `idempotent` `wait`: run the wait as normal; a timeout → skipped instead
  of failed.
- Skipped steps do **not** fail the test and do **not** trigger fail-fast.
- Keep the final verification step strict (no `idempotent`) so real
  failures in the middle of an idempotent flow still surface.

**`endpoint`** routes this step to a specific [endpoint](#endpoints). Use it to
send element targeting to one model and assertions to another.

**`wait` with `text`** waits until the page's visible text contains a
substring — no selector or LLM needed:

```toml
[[test.steps]]
kind = "wait"
target = "the success message"
text = "Welcome back"
timeout_ms = 5000
```

Set both `selector` and `text` to require both conditions. The combined wait
shares one `timeout_ms` budget.

## Failure diagnostics & artifacts

When a step fails, the runner captures the current page state and writes a
screenshot, so CI logs answer *why* the step failed instead of printing a bare
timeout:

```
  ✗ [wait] the authenticated shell — wait for app-account-shell timed out after
30000ms: The event waited for never came — page: http://127.0.0.1:8082/auth/login
(Immosai) — visible: "Email address ⏎ Password ⏎ Sign in" (30.1s)
    │ url:      http://127.0.0.1:8082/auth/login
    │ title:    Immosai — Anmeldung
    │ content:  Email address  Password  ...  Invalid credentials.
      screenshot: artifacts/account__login-and-open-account__006-wait.png
```

- **Page state** — URL, title, visible text, and any alert/error elements
  (`[role="alert"]`, `.error-message`, snackbars, …) are appended to the step
  message and printed in full to stderr.
- **Screenshots** — one PNG per failed step, written under
  `--artifacts-dir` (default `artifacts/`, env `HARNESS_ARTIFACTS_DIR`).
- **Fail fast** — by default the first failed step ends the test and the
  remaining steps are reported as skipped (no LLM budget is burned asserting
  against a page that is already known broken). Set
  `continue_on_failure = true` in `[config]` or pass `--continue-on-failure`
  to keep executing every step.
- **LLM element targeting is verified** — a response that is not a selector
  (`:not(*)`, `null`, explanations, …) fails immediately with the raw LLM
  output; a selector that matches nothing triggers one retry with feedback.
- **LLM errors are specific** — HTTP status, a truncated response-body
  snippet, and the attempt count are included, and deterministic client
  errors (401/403/404) fail fast instead of burning the retry budget.
- **Assertions always see the page** — custom `system`+`user_template`
  definitions that omit the `{content}` placeholder automatically get the
  page URL/title/content appended, so the LLM never answers "I can't
  determine that without seeing the page".

## Reporting: human- and machine-readable runs

All output flows through a single event stream. Every event (test/step
started + finished, LLM call with duration/tokens/cost, budget warning) is
rendered for humans and serialized for machines:

- **Console** — level-filtered, ASCII-safe, colors only on a TTY
  (respects `NO_COLOR`). Default shows config, per-step results with
  durations, the run summary and the cost report. Use `-q`/`-qq` to hide
  step results (then warnings too), or `-v`/`-vv` to add LLM call details
  (endpoint, model, duration, tokens, cost) and step starts.
- **NDJSON log** (`--log-file run.jsonl`) — one JSON object per event with
  a `type` discriminator and an epoch-ms `ts` field. Lossless apart from
  secret redaction: untruncated messages, ideal for CI artifact analysis:

  ```
  jq '. | select(.type == "step_finished" and .status == "failed")' run.jsonl
  jq '. | select(.type == "llm_call_finished") | {endpoint, ok, duration_ms, cost}'
  ```

- **JUnit XML** (`--junit report.xml`) — one `<testcase>` per test with a
  `<failure>` per failed step, for Jenkins/GitLab/Azure/TeamCity.
- **Perfetto trace** (`--trace run.json`) — test/step/LLM spans in Chrome
  Trace Event Format, viewable at https://ui.perfetto.dev.
- **GitHub Actions** — in CI the reporter automatically emits `::error::`
  annotations for failed steps (with the screenshot as `file=`) and appends
  a run summary to `GITHUB_STEP_SUMMARY`.

Truncation (`<truncated N chars>`) is always boundary-safe (multi-byte
UTF-8 can never panic it) and reports how much was cut; full text is
preserved in the NDJSON log.

## Secret redaction

Every sink is redacted through one chokepoint, so a leaked secret can never
make it into a log or CI report:

- **API keys** — `llm_api_key` and per-endpoint `api_key`
- **Static credentials** — Entra `auth.client_secret`, AWS
  `secret_access_key` / `session_token`
- **Sensitive header values** — values of `authorization`-family headers
  (matched case-insensitively: `authorization`, `api-key`, `x-api-key`,
  `x-auth-token`, `token`, `cookie`, …) in `llm_headers` and endpoint
  `headers`
- **Runtime-obtained tokens** — token-command and header-command output,
  Entra client-credentials and managed-identity access tokens (registered
  the moment they are fetched, so the `LlmCallFinished` event that echoes
  them is redacted too)
- **Explicit extras** — `--redact <SECRET>` (repeatable) or the
  `HARNESS_REDACT` env var (comma-separated), for secrets not present in
  the config (URL query tokens, scenario-embedded test data):

  ```
  llm-browser-testkit run scenario.toml --redact 'abc123tokenxyz'
  HARNESS_REDACT='abc123tokenxyz,xyz789tokenabc' llm-browser-testkit run scenario.toml
  ```

Secrets shorter than 6 characters are skipped when derived from config or
runtime sources so short values (e.g. `"dev"`) do not destroy log
readability; explicit `--redact` values always apply. Replacement is
exact-match and case-sensitive. Raw `eprintln!` sites that bypass the
reporter (MCP/A2A server startup banners, the `#[browser_test]` run-report
strings, the cost report) are not redacted.

## Vision assertions (screenshots)

Text/DOM evaluation cannot see *how* the page renders — overlapping elements,
clipped text, or a cookie banner covering the content are invisible to
`innerText`. Mark an endpoint as vision-capable and attach a screenshot to an
assert step to let the LLM evaluate the actual pixels:

```toml
[config]                      # optional: cap the screenshot size
screenshot_max_height = "20x"  # full page, tiled up to 20× the viewport height
screenshot_max_dimension = 1400

[config.endpoints.vision]     # MUST declare vision = true
type = "llm"
url = "https://api.openai.com"
model = "gpt-4o"
api_key = "sk-..."
vision = true                 # ← the flag
pricing = { input_per_1m_tokens = 2.50, output_per_1m_tokens = 10.00 }

[[definitions]]
name = "no_overlaps"
preset = "visual_no_overlaps"

[[test.steps]]
kind = "assert"
definition = "no_overlaps"
endpoint = "vision"
screenshot = true             # ← attach the viewport screenshot
```

How it works:

- The **full scrollable page** is captured in a single CDP call (via the
  `captureBeyondViewport` flag — nothing below the fold is skipped) and
  split into viewport-tall tiles from the top, covering at most
  `screenshot_max_height` (default `"20x"` = twenty viewports).
- Each tile is downscaled in Rust (Lanczos) so its longest edge is at most
  `screenshot_max_dimension` (default 1400) and re-encoded as quality-85
  JPEG — no page JS, deterministic, and every tile keeps 1:1 detail at its
  own depth (a single downscaled composite would lose all detail past ~4
  viewport heights). A 14400px page at 720px viewport and `"20x"` sends
  twenty 1280×720 tiles covering the whole page; `"4x"` sends four tiles
  covering 2880px. Tile count is hard-capped at 30.
- The tiles are sent as OpenAI-compatible `image_url` content parts next
  to the text prompt (which still includes the page text for context),
  ordered from the top of the page down. Token cost is effectively
  coverage ÷ viewport height — the cap bounds it. Tile count is hard-capped
  at 30 so no assertion can ever produce an unbounded request.
- `screenshot_max_height` accepts an absolute pixel count (`2880`, useful
  when you know exactly how far down a page is dynamic) or a viewport
  multiple (`"2x"`, `"20x"` — auto-follows viewport-matrix and per-test
  viewport overrides). Values below the viewport height are raised to it,
  so the visible viewport is always fully included (`0` / `"0x"` = exactly
  the viewport, the pre-full-page behavior).
- Built-in presets: `visual_no_issues`, `visual_no_overlaps`,
  `visual_text_visible` (uses `assert_text`). Custom `screenshot = true`
  prompts work too.
- A `screenshot = true` step that resolves to an endpoint without
  `vision = true` fails immediately with a clear configuration error.
- Text-only workflows are untouched: without `screenshot = true` the
  request keeps the plain string `content` shape.

See [`examples/visual-overlays.toml`](examples/visual-overlays.toml) + the
bundled [`examples/visual-test-page.html`](examples/visual-test-page.html)
fixture for a runnable demo that passes on a clean page and detects a cookie
banner overlay. The prompts of the visual presets are intentionally strict
("only fail on clearly visible, user-impacting defects") — tune them per app
if your overlay detection needs to be more or less sensitive.

## DOM layout assertions (`layout_no_issues`)

Vision models see pixels but cost money per page × viewport. For cheap,
deterministic layout coverage there is a DOM-only preset that never calls
the LLM:

```toml
[[test.steps]]
kind = "assert"
preset = "layout_no_issues"   # no endpoint, no screenshot, no tokens
```

It evaluates a geometry scan in the page and fails with the detected issues:

- **page-overflow-x** — the document is wider than the viewport
  (horizontal scrolling or a runaway element);
- **element-out-of-viewport** — a visible element that scrolling cannot
  reveal: a fixed element outside the viewport, left/negative overflow,
  right-edge overflow beyond the scrollable content, or bottom overflow
  on a page that cannot scroll down. Below-the-fold content on a tall
  scrollable page is normal flow and is NOT reported;
- **text-clipped** — content inside an `overflow: hidden` container is
  measurably larger than the box (cut-off text);
- **element-overlap** — an interactive element's center point is covered
  by a different element that would intercept the click.

Intentional stacking (off-canvas drawers, dropdowns, badges, sticky
headers) is excluded by position/relation filters. Elements whose class
matches a prefix in `[config] layout_ignore_classes` are skipped by the
fixed-element, text-clipped, and overlap checks — the default covers the
Angular CDK screen-reader helpers (`.cdk-visually-hidden`,
`.cdk-describedby-message-container`, `.cdk-overlay-container`), which
are intentionally 1x1 / off-screen:

```toml
[config]
layout_ignore_classes = ["cdk-visually-hidden", "cdk-describedby-message-container", "cdk-overlay-container"]
```
Run it after every page load — it is free, so it is also the perfect
companion for the viewport matrix below.

## Viewport matrix (mobile / tablet / desktop)

`[config.viewport_matrix]` expands **every test** in a scenario into one
variant per named viewport. Each variant overrides the browser viewport via
CDP device-metrics emulation and gets a ` — <name>` suffix on the test name;
per-test budgets apply per variant.

```toml
[config.viewport_matrix]
viewports = [
  { name = "mobile",  width = 390,  height = 844 },
  { name = "tablet",  width = 768,  height = 1024 },
  { name = "desktop", width = 1280, height = 720 },
]

[[test]]
name = "Dashboard renders"
steps = [
    { kind = "navigate", url = "/dashboard", wait_after_ms = 2000 },
    { kind = "assert", preset = "layout_no_issues" },
]
```

The above runs "Dashboard renders — mobile", "— tablet" and "— desktop",
each at its viewport, and the layout scan flags sticky overlays, off-screen
text, or covered controls per size. Use it with `screenshot = true` +
`visual_no_issues` on a vision endpoint for pixel-level checks on top.

Single-test overrides work too — any `[[test]]` may set
`viewport_width` / `viewport_height` directly, which also switches the
browser viewport via CDP for just that test:

```toml
[[test]]
name = "Narrow phone layout"
viewport_width = 320
viewport_height = 568
```

## Assertion presets

Built-in presets you can use inline or from `[[definitions]]`.

| Preset | What it checks |
|--------|---------------|
| `no_error_on_page` | No errors, stack traces, or broken UI on the page |
| `text_visible` | Specific text appears on the page (`assert_text`) |
| `element_exists` | A described UI element is present |
| `layout_no_issues` | **DOM scan, no LLM**: page overflow, elements out of viewport, clipped text, covered controls |
| `visual_no_issues` | **Screenshot**: no layout/rendering defects (overlaps, clipping, cut-off content, broken images, blank panels) |
| `visual_no_overlaps` | **Screenshot**: no elements covering other content or intercepting clicks |
| `visual_text_visible` | **Screenshot**: `assert_text` is fully visible and readable (not clipped or covered) |

Custom assertions with `prompt` send any question to the LLM:

```toml
[[test.steps]]
kind = "assert"
prompt = "Does the page have a heading that says 'Example Domain'?"
```

Custom presets with `system` + `user_template` let you define reusable assertion
logic with template variables `{url}`, `{title}`, `{content}`, `{expected_text}`,
and `{description}`. Forgetting `{content}` is no longer a problem — the page
context is appended automatically whenever the template does not reference it:

```toml
[[definitions]]
name = "text_matches"
system = "You are a QA tester."
user_template = "Does the page at {url} contain the text: {expected_text}?"
assert_text = "Welcome back"
```

## CLI reference

```
llm-browser-testkit run <scenario.toml> [<scenario2.toml> ...] [OPTIONS]
```

| Flag | Default | Description |
|------|---------|-------------|
| `--parallel` | auto | Exact max number of scenario files to run concurrently. Omit it to **auto-scale** to the machine: the runner probes available memory, learns the real per-browser footprint by trial and error, and throttles itself (never assuming a fixed size per page). See [Parallel runs](#parallel-runs). |
| `--parallel-min` | `1` | Lower bound for auto-scaling (ignored when `--parallel` is set). |
| `--parallel-max` | `0` | Upper bound for auto-scaling; `0` = unlimited (ignored when `--parallel` is set). |
| `--llm-url` | `$HARNESS_LLM_TEST_URL` or `http://localhost:8080` | OpenAI-compatible endpoint |
| `--llm-model` | `$HARNESS_LLM_TEST_MODEL` or `deepseek` | Model name |
| `--llm-api-key` | `$HARNESS_LLM_API_KEY` | API key (Bearer token) |
| `--llm-fallback-url` | `$HARNESS_LLM_FALLBACK_URL` | Fallback endpoint tried when the primary exhausts its attempts |
| `--llm-fallback-model` | `$HARNESS_LLM_FALLBACK_MODEL` | Fallback model name |
| `--llm-fallback-api-key` | `$HARNESS_LLM_FALLBACK_API_KEY` | Fallback API key |
| `--llm-header` | — | Custom header `Name:Value` (repeatable) |
| `--redact` | `$HARNESS_REDACT` (comma-separated) | Literal value redacted from all rows/sinks (repeatable) |
| `--model-param` | — | Provider param `key=value` (repeatable) |
| `--base-url` | `$HARNESS_BROWSER_BASE_URL` or `http://localhost:4200` | App under test |
| `--headless` | `true` | Run Chrome headlessly |
| `--timeout` | `60` | Seconds per action |
| `--viewport-width` | `1280` | Browser width |
| `--viewport-height` | `720` | Browser height |
| `--start-url` | `/dashboard` | First page to load |
| `--max-cost` | — | Global budget: max USD across all tests |
| `--max-tokens` | — | Global budget: max tokens across all tests |
| `--budget-enforcement` | `hard` | Budget mode: `hard` (abort) or `soft` (warn) |
| `--artifacts-dir` | `$HARNESS_ARTIFACTS_DIR` or `artifacts` | Directory for failure screenshots |
| `--continue-on-failure` | off | Keep running remaining steps after a step failure (default: fail fast) |
| `-q`, `--quiet` | off | Repeatable: hide step results (`-q`), then warnings too (`-qq`). Failures and the run summary always show |
| `-v`, `--verbose` | off | Repeatable: show LLM call details and step starts (`-v`), then everything (`-vv`) |
| `--log-file` | — | Write a machine-readable NDJSON event log (one JSON object per event, `type` + `ts` fields) |
| `--junit` | — | Write a JUnit XML report for CI systems |
| `--trace` | — | Write a Perfetto-format trace (test/step/LLM spans) |
| `--color` | `auto` | Console colors: `auto` (TTY + `NO_COLOR` aware), `always`, `never` |

CLI flags override the scenario `[config]`.

**Retry + fallback behavior:** every LLM call is retried up to
`HARNESS_LLM_CALL_ATTEMPTS` times (default 3) on transient failures
(network errors, HTTP 429/5xx, invalid JSON, and HTTP 200 with an empty
body — the gateway warm-up signature). When an endpoint still fails, the
fallback chain is tried: `--llm-fallback-url`/`--llm-fallback-model`/
`--llm-fallback-api-key` (or `$HARNESS_LLM_FALLBACK_*`) configure a single
fallback endpoint for the implicit default endpoint. Pair a cheap primary
with a more expensive, more powerful fallback — the fallback is only billed
when the primary fails. Scenarios that declare `[config.endpoints]` use
per-endpoint `fallbacks = [...]` instead (see below).

## Endpoints

Define multiple named endpoints — LLM providers, MCP servers, and A2A agents —
each with their own pricing, and route test steps to them automatically or
explicitly.

```toml
[config.endpoints.default]
type = "llm"
url = "https://api.openai.com"
model = "gpt-4o-mini"
api_key = "sk-..."
pricing = { input_per_1m_tokens = 0.15, output_per_1m_tokens = 0.60 }
default_for = ["targeting", "assertion"]

[config.endpoints.vision]
type = "llm"
url = "https://api.openai.com"
model = "gpt-4o"
api_key = "sk-..."
pricing = { input_per_1m_tokens = 2.50, output_per_1m_tokens = 10.00 }
default_for = []

[config.endpoints.db_mcp]
type = "mcp"
command = "npx"
args = ["-y", "@modelcontextprotocol/server-postgres", "postgresql://localhost/mydb"]
pricing = { per_call = 0.001 }

[config.endpoints.audit_agent]
type = "a2a"
url = "http://localhost:9090"
pricing = { per_call = 0.01 }

[[test]]
name = "Dashboard with vision"

[[test.steps]]
kind = "navigate"
url = "/dashboard"

# Use the vision endpoint just for this assertion
[[test.steps]]
kind = "assert"
preset = "no_error_on_page"
endpoint = "vision"
```

**Endpoint types:**

- `llm` — LLM chat API. Defaults to the OpenAI-compatible chat completions
  endpoint; `provider = "azure"` (Azure OpenAI) and `provider = "bedrock"`
  (AWS Bedrock, [see below](#llm-providers-azure--aws-bedrock)) are
  supported. Pricing is per-token (`input_per_1m_tokens`,
  `output_per_1m_tokens`).
- `mcp` — [Model Context Protocol](https://modelcontextprotocol.io) server.
  Launched as a subprocess via `command` + `args`. Pricing is `per_call`.
- `a2a` — [Agent-to-Agent Protocol](https://a2aprotocol.org) agent. Communicates
  via JSON-RPC over HTTP at the given `url`. Pricing is `per_call`.

**Routing:**

- `default_for` lists which task types an endpoint serves automatically
  (`targeting` for element resolution, `assertion` for assertions) — this is
  the per-task model specification: give each task type its own endpoint
  (e.g. a cheap model for targeting, a stronger one for assertions) by
  splitting `default_for` across endpoints.
- Add `endpoint = "name"` on any step or `[[test]]` group to override routing.

**Retries + fallback chains:**

- `max_attempts` (per endpoint, default 3; global env override
  `HARNESS_LLM_CALL_ATTEMPTS`) — how often a single chat completion is
  retried on transient failures before the endpoint is considered failed.
- `fallbacks = ["other_endpoint", ...]` (LLM endpoints only) — an ordered
  chain: when an endpoint exhausts its attempts, the next fallback is tried,
  and so on, until one answers. The answering endpoint is the one charged
  (per-usage cost reporting attributes the call correctly). Practical
  pattern: a cheap primary model with a more powerful, more expensive
  fallback that is only billed when the primary fails.

  ```toml
  # Cheap by default; escalate to a stronger model when the gateway is
  # down or returns garbage. Both endpoints serve both task types.
  [config.endpoints.default]
  type = "llm"
  url = "$LLM_URL"            # home gateway / cheap model
  model = "deepseek-v3"
  default_for = ["targeting", "assertion"]
  max_attempts = 5            # be patient with the local gateway
  fallbacks = ["pro"]         # escalate only after 5 attempts

  [config.endpoints.pro]
  type = "llm"
  url = "https://api.openai.com"
  model = "gpt-4.1"
  api_key = "sk-..."
  pricing = { input_per_1m_tokens = 2.00, output_per_1m_tokens = 8.00 }
  default_for = []
  ```

  Chains are cycle-guarded and deduplicated; non-LLM endpoints in a
  `fallbacks` list are skipped. When all endpoints fail, the error message
  names every endpoint and its failure.

## LLM providers: Azure & AWS Bedrock

Besides the default OpenAI-compatible API, LLM endpoints can target Azure
OpenAI and AWS Bedrock.

### Azure OpenAI

```toml
[config.endpoints.azure]
type = "llm"
provider = "azure"
url = "https://my-resource.openai.azure.com"   # resource endpoint, no path
deployment = "gpt-4o"                          # defaults to `model` when unset
api_version = "2024-10-21"                     # default when unset
api_key = "..."                                # sent as the `api-key` header
model = "gpt-4o"                               # unused by Azure; kept for pricing model names
pricing = { input_per_1m_tokens = 2.50, output_per_1m_tokens = 10.00 }
default_for = ["targeting", "assertion"]
```

Requests go to
`<url>/openai/deployments/<deployment>/chat/completions?api-version=<v>`.
With the default `api-key` auth mode the key is sent in the `api-key` header
(Azure's classic convention). Use `auth.api_key_header` to send an API key in
any custom header on any provider.

Instead of a static key you can authenticate with Entra ID — client
credentials or a managed identity (see
[LLM authentication](#llm-authentication) below).

### AWS Bedrock

```toml
[config.endpoints.bedrock]
type = "llm"
provider = "bedrock"
model = "anthropic.claude-3-5-sonnet-20241022-v2:0"
region = "eu-central-1"          # optional: overrides the chain default
profile = "staging"              # optional: named profile from ~/.aws
pricing = { input_per_1m_tokens = 3.00, output_per_1m_tokens = 15.00 }
default_for = ["targeting", "assertion"]

# Optional: explicit credentials instead of the credential chain
# [config.endpoints.bedrock.aws]
# access_key_id = "AKIA..."
# secret_access_key = "..."
# session_token = "..."          # only for temporary credentials
```

Requests are signed with SigV4 and sent to
`https://bedrock-runtime.<region>.amazonaws.com/model/<model>/converse`; the
system prompt, `temperature`, `maxTokens`, and vision screenshots map to the
Converse API (images become `image` content blocks). When no credentials are
configured, the standard AWS credential chain is used — `AWS_ACCESS_KEY_ID` /
`AWS_SECRET_ACCESS_KEY` / `AWS_PROFILE` env vars, `~/.aws/config` and
`~/.aws/credentials`, SSO, ECS and EC2 IMDS — exactly like the AWS CLI. The
resolved credentials (and region) are cached per endpoint config for the
process lifetime. Requires building with the `aws` cargo feature
(`cargo run --features aws ...`); the Docker image and release binaries
include it.

## A2A agents

Call remote A2A agents in your test scenarios as steps, or use them inside
assertion definitions for reusable agent-backed checks.

### Agent step

```toml
[config.endpoints.audit_bot]
type = "a2a"
url = "http://localhost:9090"
pricing = { per_call = 0.01 }

[[test]]
name = "Audit trail check"
steps = [
    { kind = "navigate", url = "/admin/audit" },
    { kind = "agent", agent = "audit_bot", task = "Check if user 'admin' appears in the recent audit log" },
]
```

### Agent-backed assertions

Define reusable agent assertions with `task_template`:

```toml
[[definitions]]
name = "audit_verify"
agent = "audit_bot"
task_template = "Verify that {expected_text} is true for the page at {url}"

[[test.steps]]
kind = "assert"
definition = "audit_verify"
assert_text = "the user can see the dashboard"
```

Template variables available: `{url}`, `{title}`, `{content}`, `{expected_text}`,
`{description}`, `{task}`.

## Run as an A2A agent

Enable the `a2a-server` feature to expose the framework as an A2A agent that
other agents or orchestrators can call. The server listens on a port and accepts
`tasks/send` JSON-RPC requests.

```toml
[config.a2a_server]
enabled = true
port = 3100
```

```bash
# Build and run with the a2a-server feature
cargo run --features a2a-server -- run scenario.toml --agent-port 3100
```

Or via CLI without modifying the TOML:

```bash
llm-browser-testkit run scenario.toml --agent-port 3100
```

### Docker deployment

```bash
docker build -t llm-browser-testkit .
docker run --rm \
  -p 3100:3100 \
  -v "$(pwd)/scenario.toml:/scenario.toml:ro" \
  -e HARNESS_LLM_TEST_URL=https://api.openai.com \
  -e HARNESS_LLM_TEST_MODEL=gpt-4o-mini \
  -e HARNESS_LLM_API_KEY=sk-... \
  llm-browser-testkit run /scenario.toml --agent-port 3100
```

A `Dockerfile` is included in the repository — it uses a multi-stage build with
Alpine and Chromium. The image is built with `--all-features`, so all LLM
providers (including Azure and AWS Bedrock) are available out of the box.

The image can be used as a GitHub Actions / Forgejo job container
(`container.image`): its entrypoint keeps the container alive (`tail -f
/dev/null`) so the runner can `docker exec` job steps into it, and prints
the harness version on boot so CI logs always show which image ran. A bare
`docker run <image>` boots the (idle) container; run a scenario in one
command by appending the args, e.g.
`docker run <image> run /scenario.toml --agent-port 3100`. `docker run
<image> sh` drops into a debugging shell.

The published image also ships the `aws` and `az` CLIs (useful for
`auth.mode = "token-command"`, e.g. `az account get-access-token`). Leaner
variants can be built locally — both are opt-in build args, off by default:

```bash
docker build -t llm-browser-testkit:aws --build-arg ENABLE_AWS_CLI=true .
docker build -t llm-browser-testkit:azure --build-arg ENABLE_AZURE_CLI=true .
```

## MCP tools

Call MCP server tools directly from test steps to query databases, read files,
or invoke any tool an MCP server exposes.

```toml
[config.endpoints.db]
type = "mcp"
command = "npx"
args = ["-y", "@modelcontextprotocol/server-postgres", "postgresql://localhost/mydb"]
pricing = { per_call = 0.001 }

[[test]]
name = "Database smoke test"
steps = [
    { kind = "navigate", url = "/dashboard" },
    { kind = "mcp", server = "db", tool = "query", args = { sql = "SELECT count(*) FROM users" } },
    { kind = "assert", preset = "no_error_on_page" },
]
```

MCP servers are launched as subprocesses via the configured `command` and `args`.
The framework handles the MCP initialize handshake, tool listing, and invocation
automatically.

## MCP server exposure

Enable the `mcp-server` feature to expose the framework as an MCP server so
other tools can invoke it remotely.

```toml
[config.mcp_server]
enabled = true
port = 3000
```

```bash
cargo run --features mcp-server -- run scenario.toml
```

When enabled, other MCP clients can call tools like `run_scenario` and
`get_page_state` on port 3000.

## Cost tracking & budgets

Every LLM call, agent invocation, and MCP tool call is tracked. After the run
completes, a cost report is printed with per-test and per-endpoint breakdowns.

### Per-test default budget

```toml
[config.budgets.per_test_default]
max_cost = 1.0
max_tokens = 100_000
max_calls = 50
enforcement = "hard"
```

### Per-test override

```toml
[[test]]
name = "Expensive test"
budget = { max_cost = 2.0, max_tokens = 200_000, enforcement = "soft" }
```

### Global budget

```toml
[config.budgets.global]
max_cost = 5.0
max_tokens = 500_000
enforcement = "hard"
```

### Enforcement

| Mode | Behavior |
|------|---------|
| `hard` | Abort the test or run immediately when budget is exceeded |
| `soft` | Print a warning but continue executing remaining steps |

### CLI budgets

```bash
llm-browser-testkit run scenario.toml --max-cost 10.0 --max-tokens 1000000 --budget-enforcement soft
```

### Sample report output

```
-------------------------------
  COST REPORT
-------------------------------
  Test: "Homepage loads" — $0.0123 | 1,234 tokens (800 in / 434 out, 120 cached, 40 cache write) | 4 calls
    models: deepseek
    endpoint.default:     4 calls,     800 in /     434 out (120 cached, 40 cache write),   1,234 tokens, $0.0123
      models: deepseek
  Test: "Dashboard smoke" — $0.0891 | 4,567 tokens (3,000 in / 1,567 out, 0 cached, 0 cache write) | 6 calls
    models: deepseek, gpt-4o
    endpoint.vision:      2 calls,   2,000 in /   1,000 out (0 cached, 0 cache write),   3,000 tokens, $0.0450
      models: gpt-4o
    endpoint.default:     3 calls,   1,000 in /     567 out (0 cached, 0 cache write),   1,567 tokens, $0.0441
      models: deepseek
    endpoint.audit_bot:   1 call,        0 in /       0 out (0 cached, 0 cache write),       0 tokens, $0.0000
-------------------------------
  GLOBAL SUMMARY
    Total cost:         $0.1014
    Total tokens:       5,801
    Total input:        3,800
    Total output:       2,001
    Total cached input: 120
    Total cache write:  40
    Total calls:        10
    Models used:        deepseek, gpt-4o
-------------------------------
```

Prompt-cache token counters are reported per provider. **Cache read** tokens
(`cached`) are prompt tokens served from a cache; **cache write** tokens are
prompt tokens written into a cache. The counters are captured from
OpenAI/Azure/OpenRouter (`prompt_tokens_details.cached_tokens` /
`cache_write_tokens`), Anthropic-style responses (`cache_read_input_tokens` /
`cache_creation_input_tokens`), DeepSeek (`prompt_cache_hit_tokens`), Google
Gemini (`usageMetadata.cachedContentTokenCount`), and AWS Bedrock Converse
(`cacheReadInputTokens` / `cacheWriteInputTokens` / `cacheDetails`). Semantics
differ by provider: for OpenAI-style responses the cached counts are a
**subset** of the input tokens, while for Anthropic/Bedrock the cache counters
are reported **in addition** to `inputTokens`. Only input (prompt) caching
exists — no provider exposes a cached-output metric.

### Prompt caching: enabled by default

Caching is **on by default** and only sends request-side markers where a
provider requires them:

- **AWS Bedrock** gets a `cachePoint` block after the system message.
- **Anthropic-style OpenAI-compatible models** (model name contains
  `claude`/`anthropic`, e.g. via OpenRouter) get a
  `cache_control: {"type":"ephemeral"}` block on the system message.
- OpenAI, Azure, Groq, xAI and DeepSeek cache automatically; no marker is
  sent.

If a Bedrock model does not support prompt caching, the `cachePoint` block is
rejected with HTTP 400 and the call is retried once **without** it, so caching
never breaks an otherwise valid call.

Disable it globally or per endpoint:

```toml
[config]
cache = false            # global default for all endpoints

[config.endpoints.default]
cache = false            # or just this endpoint
```

### Prompt-caching support matrix

`provider = "openai"` is OpenAI-compatible, so any endpoint that speaks the
chat-completions shape (OpenRouter, Groq, xAI, DeepSeek, Together, Fireworks,
local vLLM/llama.cpp, …) is reachable and uses the shared extraction below.
Provider-native-only APIs (Google Gemini `generateContent`, Anthropic
`/v1/messages`) are only parsed via their cache-field fallbacks, not as a full
request/response protocol.

| Provider / endpoint | Cache read field | Cache write field | Enabling | Read/write token semantics | Parsed by this library |
| --- | --- | --- | --- | --- | --- |
| OpenAI (Chat Completions & Responses) | `prompt_tokens_details.cached_tokens` | `prompt_tokens_details.cache_write_tokens` (GPT-5.6+) | automatic (implicit) or explicit `prompt_cache_breakpoint` | both are **subsets** of `prompt_tokens`/`input_tokens` | read ✅, write ✅ |
| Azure OpenAI | `prompt_tokens_details.cached_tokens` | `prompt_tokens_details.cache_write_tokens` (GPT-5.6+) | automatic; explicit breakpoints GPT-5.6+ | subsets of `prompt_tokens` | read ✅, write ✅ |
| OpenRouter | `prompt_tokens_details.cached_tokens` | `prompt_tokens_details.cache_write_tokens` | `cache_control` (sent for Claude models) | subsets of `prompt_tokens` | read ✅, write ✅ |
| Groq | `prompt_tokens_details.cached_tokens` | — (none) | automatic, GPT-OSS models only | subset of `prompt_tokens` | read ✅, write n/a |
| xAI (Grok) | `prompt_tokens_details.cached_tokens` | — (none) | automatic | subset of `prompt_tokens` | read ✅, write n/a |
| DeepSeek | `prompt_cache_hit_tokens` | — (none; `prompt_cache_miss_tokens` is the uncached part) | automatic | subset of `prompt_tokens` | read ✅, write n/a |
| Anthropic Messages (native) | `cache_read_input_tokens` | `cache_creation_input_tokens` | `cache_control` block/auto | **additional** to `input_tokens` | fields ✅ (not a native transport) |
| AWS Bedrock Converse | `cacheReadInputTokens` | `cacheWriteInputTokens` (+ `cacheDetails` per-TTL) | `cachePoint` block (sent by default) | **additional** to `inputTokens` | read ✅, write ✅ (`cacheDetails` summed into cache write) |
| Google Gemini `generateContent` (native) | `usageMetadata.cachedContentTokenCount` | — (none) | explicit cached content / implicit | separate counter | read ✅ (via `usageMetadata` fallback) |
| Google Vertex AI (Anthropic models) | `cache_read_input_tokens` | `cache_creation_input_tokens` | `cache_control` | additional to `input_tokens` | fields ✅ (via fallback) |

Notes:

- Cache **writes** are only meaningful for providers with explicit caching and
  cache-write billing. OpenAI-family and Groq/xAI/DeepSeek auto-cache and
  expose reads only; their write counter stays `0`.
- Because write/read counters are subsets for OpenAI-style providers but
  additive for Anthropic/Bedrock, `Total input` is the provider-reported input
  count: for OpenAI-style providers it already includes cache reads/writes,
  while for Anthropic/Bedrock the true processed prompt is
  `input + cached + cache write`.

### Cache-aware cost pricing

Costs are cache-aware by default: cache reads are billed at
`input_price_per_1m × 0.1` and cache writes at `input_price_per_1m × 1.25`
(industry-standard multipliers), with cache tokens treated as a subset or as
additive to input depending on the provider. Override or disable:

```toml
[config.endpoints.default.pricing]
input_per_1m_tokens = 3.0
output_per_1m_tokens = 15.0
cached_input_per_1m_tokens = 0.30   # optional, overrides the 0.1x multiplier
cache_write_per_1m_tokens = 3.75    # optional, overrides the 1.25x multiplier
cache_read_multiplier = 0.1         # optional
cache_write_multiplier = 1.25       # optional
cache_pricing = false               # bill every prompt token at the input price
```

### Automatic pricing lookup

Pricing lookup is **on by default** (`pricing_source = "auto"`). Where a
provider exposes **exact**, machine-readable prices, the lookup runs once at
startup and fills only the pricing fields you left unset (explicit values
win). A lookup failure is non-fatal: the run continues with the configured
pricing. `auto` uses the Bedrock Price List for Bedrock endpoints and the
OpenRouter models API for `openrouter.ai` URLs; all other providers are a
no-op. When auto is on but the provider exposes no exact source (OpenAI,
Azure, Google, Groq, xAI, DeepSeek), the endpoint is reported with a warning
and its `pricing_source` is set to `off`.

```toml
# auto is the default — no pricing_source needed:
[config.endpoints.bedrock]
provider = "bedrock"
model = "us.anthropic.claude-3-5-sonnet-20241022-v2:0"
# region comes from [..aws].region, else AWS_REGION, else us-east-1

[config.endpoints.openrouter]
model = "anthropic/claude-3.5-sonnet"
# URL host openrouter.ai -> OpenRouter pricing

# Force a source, or turn the lookup off:
[config.endpoints.other]
pricing_source = "bedrock"      # "openrouter" | "off" | "none" | "disabled"
```

`openrouter` fetches exact per-token `prompt`, `completion`, `input_cache_read`
and `input_cache_write` prices from `GET https://openrouter.ai/api/v1/models`.

`bedrock` fetches the AWS Price List at startup and merges two public offers
for the endpoint's region — `AmazonBedrock` (Nova, Llama, Mistral, DeepSeek,
…) and `AmazonBedrockFoundationModels` (Anthropic Claude, Cohere, …) — reading
the exact input/output/**cache-read**/**cache-write** price for the configured
model. Standard on-demand, in-region prices are used; `batch`, `flex`,
`priority`, `global`, latency-optimized, provisioned-throughput and custom-model
tiers are excluded. The model id is normalized (inference-profile and provider
prefixes, dates and `:0` revisions are stripped) and matched to the catalog
name, e.g. `us.anthropic.claude-3-5-sonnet-20241022-v2:0` →
`Claude 3.5 Sonnet v2`.

| Provider | Exact public price API? | Status in this library |
| --- | --- | --- |
| OpenRouter | Yes — `/api/v1/models` (no auth) | ✅ `pricing_source = "openrouter"` |
| AWS Bedrock | Yes — Price List `AmazonBedrock` + `AmazonBedrockFoundationModels` per region, incl. cache read/write | ✅ `pricing_source = "bedrock"` |
| Azure OpenAI | Partly — public Retail Prices API, but model→meter mapping is heuristic | ❌ not implemented |
| OpenAI, Groq, xAI, DeepSeek | No public price API | ❌ configure `pricing` manually |
| Google Gemini / Vertex | Pricing page only; Cloud Billing Catalog needs a key and lacks Gemini dev prices | ❌ configure `pricing` manually |



## Parallel runs

Pass several scenario files to run them concurrently. Each file executes on
its **own isolated browser** (a separate Chrome process), so cookies,
localStorage, and other session state can never leak between files. The
steps inside each file still run sequentially on that file's browser.

```console
llm-browser-testkit run checkout.toml search.toml cart.toml
```

All files in one batch share the same CLI overrides and report as a single
run: one `RunStarted`/`RunFinished` event, one merged cost report, and a
combined exit code (non-zero if any file failed).

At the end of a batch the cost report prints a **`PER-FILE SUMMARY`** (each
file's cost, tokens and calls) followed by the combined `COST REPORT` whose
`GLOBAL SUMMARY` totals tokens, input/output/cached/cache-write tokens and cost
across **all** files:

```console
-------------------------------
  PER-FILE SUMMARY
-------------------------------
  checkout.toml: $0.0310 | 3,100 tokens (2,000 in / 1,100 out, 400 cached, 0 cache write) | 7 calls
  search.toml: $0.0080 |   800 tokens (  600 in /   200 out,   0 cached, 0 cache write) | 2 calls
-------------------------------
  COST REPORT
  ...
  GLOBAL SUMMARY
    Total cost:         $0.0390
    Total tokens:       3,900
    Total input:        2,600
    Total output:       1,300
    Total cached input: 400
    Total cache write:  0
    Total calls:        9
    Models used:        deepseek
-------------------------------
```

### Auto-scaling concurrency

By default the runner **auto-scales** to the machine rather than assuming
each browser costs a fixed amount of memory, and it is parallel out of the
box:

1. It probes available physical memory once (Linux `free`, macOS
   `sysctl`/`vm_stat`).
2. Around each file it measures how much memory one browser actually holds
   (`available` before minus after), folding that into a learned, clamped
   per-browser footprint.
3. After each success it raises the concurrency limit toward
   `available ÷ footprint`; when memory can't be measured it ramps up one
   browser at a time (up to a default ceiling of 8).
4. It **guards** launches: a new browser is not started while less than an
   absolute 256 MiB is free, or when there is no room for one more browser
   of the learned footprint. The guard uses absolute free bytes — never a
   fraction of `total` memory — so it behaves correctly inside VMs and
   containers, where `total` often reports the host's much larger RAM.
5. If a launch fails with an out-of-memory-style error it **halves the
   limit** and **retries the file** with backoff (up to 3 attempts) before
   reporting it failed.

The bounds are configurable: `--parallel-min N` / `--parallel-max N`
(default `1` / unlimited). Pin an exact count with `--parallel N`, which
disables auto-scaling (the memory guard and retries still apply).

### Controlling which files may overlap

By default every file has its own implicit group, so distinct files run in
parallel. When several files touch the **same shared backend state** (e.g.
they all mutate the same checkout ledger) and would interfere if run at the
same time, give them a concurrency group — files that declare the same group
are never executed concurrently:

```toml
[config]
concurrency_group = "checkout"   # files sharing this group never overlap
```

Files with no `concurrency_group` (or different groups) may still run at the
same time, subject to `--parallel`. The limit only caps how many files run at
once; it does not schedule the order of files within a group.

## How it works

Four pieces:

1. **Chrome** — launched via the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/)
   (`headless_chrome` crate). It navigates, clicks, types, and extracts page
   content.

2. **LLM** — any OpenAI-compatible API. Used in two places:
   - **Element targeting**: when a step says `target = "the login button"`, the
     runner sends the page's interactive elements to the LLM and asks for a CSS
     selector.
   - **Assertions**: the runner sends page content to the LLM with a QA prompt
     and expects `PASS` or `FAIL: <reason>`.

3. **A2A + MCP** — connect to remote agents via the Agent-to-Agent Protocol
   and to MCP servers for tool-calling. Both are first-class step kinds.

4. **TOML scenarios** — declarative test files. No code, no CSS selectors
   required. Just describe what you want in English.

```
TOML file  →  CLI runner  →  Chrome (CDP)  →  LLM API
                                        →  A2A agent
                                        →  MCP server
```

## Use as a library

```toml
[dependencies]
llm-browser-testkit = { version = "0.1", features = ["macros", "mcp-server"] }
```

```rust
use llm_browser_testkit::runner::ScenarioRunner;
use llm_browser_testkit::scenario::Scenario;

let scenario: Scenario = toml::from_str(&contents)?;
let runner = ScenarioRunner::new(scenario.config.clone(), scenario.definitions);
let report = runner.run(&scenario.test)?;

println!("Passed: {}, Failed: {}", report.tests_passed, report.tests_failed);

// Access cost/usage data
let usage = runner.usage_tracker();
let global = usage.global_snapshot();
println!("Total cost: ${:.4}", global.total_cost);

// Print the cost report
llm_browser_testkit::reporting::print_report(
    &usage.per_test_snapshots(),
    &global,
);
```

### Macros: `#[browser_test]` in `cargo test`

Enable the `macros` feature to write browser tests directly in your Rust test
modules:

```toml
[dev-dependencies]
llm-browser-testkit = { version = "0.1", features = ["macros"] }
```

```rust,ignore
use llm_browser_testkit::browser_test;
use llm_browser_testkit::browser_test_inline;

// Run a TOML scenario file
browser_test!(homepage => "tests/homepage.toml");

// Inline small scenarios
browser_test_inline!(hello, r#"
[config]
base_url = "https://example.com"

[[test]]
name = "hello"

[[test.steps]]
kind = "navigate"
url = "/"

[[test.steps]]
kind = "assert"
preset = "no_error_on_page"
"#);
```

Tests auto-skip when no LLM endpoint or Chrome is available — safe to include in
every CI run. They only execute with real `PASS`/`FAIL` when infrastructure is
present.

## LLM authentication

The runner supports API keys, Entra ID tokens, token commands, and custom
(static or command-produced) headers — per endpoint.

**API key** (default `api-key` mode): sent as `Authorization: Bearer <key>`
on OpenAI-compatible endpoints, as the `api-key` header on Azure. Set
`auth.api_key_header` to use a different header name on any provider:

```toml
[config.endpoints.api]
type = "llm"
url = "https://api.openai.com"
api_key = "sk-..."
model = "gpt-4o"

[config.endpoints.api.auth]
mode = "api-key"                # default; can be omitted
api_key_header = "X-Api-Key"    # send the key here instead of Authorization
```

**Custom headers** — static `headers` and per-call `header_commands`
(provider-agnostic; the command's first stdout line becomes the header value):

```toml
[config.endpoints.prod]
type = "llm"
url = "https://api.example.com"
model = "gpt-4o"
headers = { "X-Org-ID" = "acme" }            # static

[config.endpoints.prod.header_commands]
X-M2M-Token = "kubectl exec tokenizer -- token"   # dynamic, per call
```

**Token commands** (`mode = "token-command"`): run any program, its stdout
(first line) becomes the bearer token. Works with any CLI that prints a
token — Azure CLI, Vault, ...:

```toml
[config.endpoints.azure]
type = "llm"
provider = "azure"
url = "https://my-resource.openai.azure.com"
model = "gpt-4o"

[config.endpoints.azure.auth]
mode = "token-command"
token_command = "az account get-access-token --resource https://cognitiveservices.azure.com --query accessToken -o tsv"
cache_ttl_secs = 240            # reuse the token this long (default 300)
```

**Entra ID client credentials** (`mode = "entra-client-credentials"`): the
OAuth 2.0 client-credentials grant against
`https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token`; the token is
cached until its server-issued expiry, then refreshed automatically:

```toml
[config.endpoints.azure.auth]
mode = "entra-client-credentials"
tenant_id = "<tenant-or-uuid>"
client_id = "<app-registration-client-id>"
client_secret = "<client-secret>"
scope = "https://cognitiveservices.azure.com/.default"   # default
```

**Entra ID managed identity** (`mode = "entra-managed-identity"`): fetches a
token from the Azure IMDS endpoint — zero credentials in the config; works on
Azure VMs, App Service, and ACI with a system-assigned identity:

```toml
[config.endpoints.azure.auth]
mode = "entra-managed-identity"
```

Token acquisition is cached process-wide per auth configuration, so a test
run authenticates once instead of on every LLM call.

Via CLI / env, the runner still supports plain keys and headers:

```bash
llm-browser-testkit run tests.toml \
  --llm-api-key sk-... \
  --llm-header "X-Org-ID:acme" \
  --llm-header "X-Project:qa"
```

```bash
export HARNESS_LLM_API_KEY=sk-...
export HARNESS_LLM_HEADERS='{"X-Org-ID":"acme","X-Project":"qa"}'
```

## License

Apache-2.0 OR MIT