# OpenRouter E2E acceptance
The gated, real-network acceptance suite in `tests/openrouter_e2e.rs` is the
legacy chat-tool acceptance suite. It does not exercise the Skills or Agents
tools. It drives real MCP servers, the real protocol lifecycle, and a real
frontier model over the real OpenRouter API.
## What the suite does
1. Spawns the filesystem, fetch, memory, and shell servers in a disposable environment and
connects via the modern `server/discover` lifecycle, asserting the
negotiated protocol version (`2026-07-28`), server identity
(`mcp-filesystem`, `mcp-fetch`, `mcp-memory`, `mcp-shell`),
capabilities, and instructions.
2. Derives every OpenRouter function schema from the runtime MCP `Tool`
definitions through a single deterministic normalizer/validator
(`tests/openrouter/schema.rs`) — no duplicated handwritten schemas.
3. Runs one forced `assistant -> tool -> assistant` roundtrip per case:
the model must call exactly the intended tool with exactly the intended,
schema-valid arguments; the MCP tool executes; the result returns with
the matching `tool_call_id`; tools are resent; the final answer must be
bounded and non-empty. An independent programmatic oracle checks the MCP
result and fixture side effects (disk content, JSONL state, content
blocks).
4. Consumes the `fetch` prompt and the `memory://knowledge-graph` resource
through one bounded real request each, with the content fed into the
model.
5. Enforces coverage: `assert_coverage` requires exact set equality between
the case catalog and the live tool inventory, plus per-parameter
coverage (provided for every parameter; additionally omitted for
optional parameters). Adding a tool or parameter without updating the
catalog fails the suite.
Fetch correctness never depends on the public internet: every fetch case
uses a deterministic local `tiny_http` fixture (including robots.txt
modes). Shell cases use an isolated temp cwd and this test binary as a
deterministic helper; no destructive commands are ever executed.
## Model and bounds
- Endpoint: `POST https://openrouter.ai/api/v1/chat/completions`, Bearer
auth from `OPENROUTER_API_KEY` (environment only).
- Required default model alias: `~deepseek/deepseek-v4-flash-latest`
(OpenRouter `~` = latest of the family). `OPENROUTER_MODEL` is accepted
as an override for diagnostics, but the acceptance run must use the
default alias; the response `model` field reports the concrete resolved
model.
- Forced tool shape `{"type":"function","function":{...}}`;
`parallel_tool_calls: false`, `stream: false`, `temperature: 0`.
- Per-request: timeout 45 s, response body ≤ 1 MiB, `max_tokens` 512 per
tool call and 200 per final answer/consumption request.
- Retry policy: at most 2 attempts per request, retrying only transport
errors, 429s, 5xx, and `Retry-After` responses (backoff capped at 5 s).
In addition, a whole roundtrip is retried up to 2 times for
model-generation flakes only: a model-echo deviation (dropped/rewritten
argument — the argument guard runs again and still blocks before any MCP
execution), truncated (unparseable JSON) arguments, a `length` finish
reason without the forced tool call, or an empty final answer cut off by
`length`. Benign default padding (schema-declared defaults, the natural
zero value of an optional property, and the shell `cwd` context field) is
stripped before comparison. Oracle failures, schema failures, and server
failures are never retried.
- Suite budget: 14 minutes 30 seconds; a hard abort still reports all
failures collected so far.
- Tool results fed back to the model are capped at 4000 characters per call.
## How to run it
```bash
OPENROUTER_API_KEY=<key in the environment> \
env -u OPENROUTER_MODEL cargo test --test openrouter_e2e \
-- --ignored --nocapture --test-threads=1
```
- The suite is **ignored by default** — ordinary `cargo test` runs stay
offline, secret-free, and cost-free.
- Invocation without `OPENROUTER_API_KEY` fails clearly; the gating cannot
silently skip.
- The key is read from the environment at runtime and is never logged,
printed, or written anywhere. The summary prints only case metadata,
request IDs, resolved model names, token usage, and elapsed times.
## Most recent verified acceptance run
The following numbers characterize the latest historical run of the suite
(2026-08-14). They are a record of that run, not a guarantee for future
runs: model names, token usage, and timing vary with OpenRouter's routing
and the resolved model version. The **repeatable guarantees** are the
harness invariants above (coverage assertions, argument guard, oracles,
bounds, retry policy).
| Requested model alias | `~deepseek/deepseek-v4-flash-latest` |
| Actual resolved model | `deepseek/deepseek-v4-flash-0731` |
| Cases executed | 66 (29 filesystem, 16 memory, 11 fetch, 10 shell) |
| MCP tool calls | 66 |
| Full roundtrips | 66 |
| Real OpenRouter HTTP attempts | 135 (including retries) |
| Transport retries | 1 |
| Prompt/resource consumption requests | 2 (fetch prompt, memory resource) |
| Tokens (in / out) | 54,878 / 11,647 |
| Total elapsed | 404 s |
All 66 cases passed with the programmatic oracles; the fetch prompt and the
memory resource were consumed successfully in real requests.
## Metrics captured
Per case: case ID, server, tool, status (ok/deviation/error), request IDs,
actual (resolved) model, token usage, elapsed, MCP status, retries.
Aggregate: real HTTP attempt count including retries, MCP tool calls, full
roundtrips, prompt/resource consumption requests, tokens in/out, retries.
Never printed: API key, authorization headers, raw response bodies.
The full per-tool case matrix is maintained in the [Coverage matrix](/coverage).