Expand description
The agent/QA tool API — a localhost tool surface on a Bevy app so LLM agents can inspect
game state, capture screenshots, and drive input for automated QA/playtesting. Ported from
PROTOTYPE_19’s dev/tool_api (ADR 0009) and generalized into a reusable plugin.
Three layers:
- BRP (the Bevy Remote Protocol,
bevy_remote) as the data layer: JSON-RPC 2.0 over HTTP on127.0.0.1:15702. The built-in methods (bevy/query,bevy/get_components,bevy/list+watch,bevy/spawn, …) expose the whole reflected ECS for free. - Custom BRP methods as the game tools:
game/state(an optional host-registered snapshot hook — agents work better with a small structured view than raw ECS dumps),game/screenshot+game/screenshot/get(capture via bevy’sScreenshot→ PNG on disk → base64 on poll),game/gamepad(mocksbevy_input::gamepad::Gamepad‘s own button/axis state on a synthetic gamepad entity, so injected state flows through input crates’ real binding resolution — dead zones, shared-device ownership, per-context device selection — exactly like a human’s controller),game/keyboard(mocksButtonInput<KeyCode>directly — seegame/keyboard), andgame/mouse(mocksButtonInput<MouseButton>plus realMouseMotion/MouseWheelevents, and drivesbevy_picking’s ownPointerInputpipeline for cursor position and UI clicks — seegame/mouse’s doc comment, including a real gotcha found by testing:AccumulatedMouseMotion/AccumulatedMouseScrollcan’t be set directly, only injected as events).game/uiis the vision aid that pairs with all of this: an accessibility-tree dump of labeled rects + text in screenshot pixel space, so the model reads rows instead of OCR-ing pixels. - An in-process MCP server (
rmcp, Streamable HTTP on127.0.0.1:15710, stateless mode) whose tools proxy to the BRP methods over loopback HTTP — the MCP layer owns only the protocol surface (tool listing + schemas), never theWorld(the handlers are async and run outside Bevy’s world; allWorldaccess stays in BRP’s systems).read_guideserves the agent guides bundled into the binary at compile time — an agent connected to the MCP server can read the playtesting playbook with zero setup.
Never enable this in player-facing builds: it is a debug/QA tool surface and BRP is unauthenticated by design — localhost bind only.
§Extending with game-specific tools
Two independent extension points, both available to the host app at any time (before or
after the plugin is added, in any plugin’s build):
-
Custom BRP methods — attach a system to the
bevy::remote::RemoteMethodsresource. Handlers run in the main world with&mut Worldaccess, so they can read game state directly:fn register_my_methods(app: &mut App) { register_game_method(app, "game/my_state", my_game_state); }Methods can declare a precondition (
register_game_method_with_precondition) — a&World(+ intended params) check surfaced by the{prefix}/plan_checkpre-flight method and its MCPplan_checktool, so an agent can verify a multi-step sequence would pass before sending it:register_game_method_with_precondition( &mut app, "game/my_state", Some(std::sync::Arc::new(in_lobby) as PreconditionFn), my_game_state, ); -
Custom MCP tools — pass
HarnessTools viaMcpHarnessConfig::extra_tools. The callback runs on the MCP server thread with parsed arguments and aBrpClient; the conventional shape proxies to a custom BRP method like the one above:HarnessTool::new( "my_state", "Game-specific state snapshot.", |client: BrpClient, args: MyToolArgs| async move { let mut params = serde_json::json!({}); if args.detailed { params["detailed"] = serde_json::json!(true); } client.call("game/my_state", params).await }, )(The host crate needs
serde,schemars, andserde_jsonin its[dependencies]for the argument struct derives — the harness re-exportsBrpClientandHarnessToolitself.)
§Usage
use bevy::prelude::*;
use bevy_mcp_harness::{BevyMcpHarnessPlugin, OffscreenMode};
App::new()
.add_plugins(BevyMcpHarnessPlugin::default()) // windowed app: tool surfaces only
// Headless agent host (no window): render into a 1280×800 offscreen texture instead.
// .add_plugins(BevyMcpHarnessPlugin { config: McpHarnessConfig {
// offscreen: OffscreenMode::Owned(UVec2::new(1280, 800)),
// ..McpHarnessConfig::from_env() } })
.run();Re-exports§
pub use headless::HeadlessUiCameraBootstrap;pub use headless::NoRenderMode;pub use headless::OffscreenRenderTarget;pub use mcp::BrpClient;pub use mcp::HarnessTool;pub use schemars;pub use serde_json;
Modules§
- brp
- The custom BRP methods — the game tools. Handlers run in the main world with
&mut Worldaccess (registered as plain systems intobevy::remote::RemoteMethods); the MCP layer proxies to them over loopback HTTP. Ported from PROTOTYPE_19’sdev/tool_api.rs, minus its game-specific methods (game/inputaction mocks,game/select,game/trigger,game/levels,game/select_level— all bound to that prototype’s own crates). - headless
- Headless-mode rendering support: the shared offscreen texture every camera renders into,
the camera retargeting that makes window-less apps produce frames, and the render-less
(
no_render) shim that keeps UI layout working without any render app. Ported from PROTOTYPE_19’scontrols/camera.rs. - mcp
- The in-process MCP server (rmcp, Streamable HTTP, stateless) — the protocol surface whose
tools proxy to the BRP methods in
crate::brpover loopback HTTP. The MCP layer owns only the protocol surface (tool listing + schemas), never theWorld(the handlers are async and run outside Bevy’s world; allWorldaccess stays in BRP’s systems).
Structs§
- Bevy
McpHarness Plugin - The MCP + Agent playtesting harness. Adds the BRP server (unless the host already did),
registers the
game/*custom methods, runs the in-process MCP server on a background thread, and — configured for headless use — owns the offscreen render target machinery. - McpHarness
Config - Harness configuration, inserted as a resource at plugin build.
Defaultis the plain windowed setup (tool surfaces on the default ports, screenshots into<cwd>/mcp_harness/screenshots/);McpHarnessConfig::from_envapplies the same CLI flags the prototype used (--brp-port N,--mcp-port N,--no-render).
Enums§
- Offscreen
Mode - The headless rendering mode — see
McpHarnessConfig::offscreen.
Constants§
- DEFAULT_
MCP_ PORT - The MCP surface’s default port — NOT 15703, which is
bevy_remote’s render-subapp BRP port (DEFAULT_RENDER_PORT, active wheneverbevy_renderruns): binding the MCP listener there makes the render app’s BRP bind fail and the main BRP pipeline hang in release builds. - DEFAULT_
OFFSCREEN_ SIZE - The offscreen capture target’s default size (Steam Deck 800p) for
McpHarnessConfigusers that want the proven headless geometry.
Functions§
- isolated_
screenshots_ dir - Appends the fleet-testing isolation rule to a screenshots base directory: a non-default
brp_portcaptures into a per-clientscreenshots/client-<port>/subdirectory instead of the base (a shared directory would cross-contaminategame/screenshot/getpolls between concurrently running clients). Hosts that keep their own screenshots convention use this with their base directory so they get the isolation rule for free. - register_
game_ method - Registers a custom BRP method (the harness’s own
game/*methods are registered the same way):systemruns in the main world with&mut Worldaccess whenever an agent calls<name>over BRP. Must be called afterBevyMcpHarnessPlugin(or any otherRemotePluginadder) so theRemoteMethodsresource exists — frommain, or from a later plugin. Returns the registered system’s id. - register_
game_ method_ with_ precondition register_game_methodwith a declaredPreconditionFn— surfaced by the{prefix}/plan_checkpre-flight method (and its MCPplan_checktool) so an agent can check which calls in an intended sequence would fail, and why, before sending them. Advisory only: the method’s own checks remain the source of truth at call time.
Type Aliases§
- Clickable
Fn - The host’s clickable-UI convention for
game/ui: returntruewhenentityis a button/interactive node under an interaction convention the harness can’t know (an HTML-markup UI’s click hooks, say).bevy_ui::Interactionholders are always reported clickable; this hook adds to them. - Precondition
Fn - A declared precondition for a BRP method, checked by
plan_check(pre-flight) before the agent burns a call on a method that would fail.worldis the app’s world;paramsis the params object the agent intends to send (Nonewhen the call carries none) — event-shaped methods key their requirements off it.Err(reason)is the human-readable failure the agent would hit. - State
Snapshot Fn - The host-registered
game/statesnapshot: everything an agent needs to reason about this game (app state, the player’s entity/position/health, whatever matters here). Called with&mut Worldfrom the BRP handler thread-side systems; return a JSON object. Fused into everygame/screenshot/getresponse (and its.jsonsidecar) so a screenshot arrives with its ground-truth state attached.