Skip to main content

Crate bevy_mcp_harness

Crate bevy_mcp_harness 

Source
Expand description

The agent/QA tool API — a localhost tool surface on a Bevy app so LLM agents can inspect game state, capture screenshots, and drive input for automated QA/playtesting. Ported from PROTOTYPE_19’s dev/tool_api (ADR 0009) and generalized into a reusable plugin.

Three layers:

  1. BRP (the Bevy Remote Protocol, bevy_remote) as the data layer: JSON-RPC 2.0 over HTTP on 127.0.0.1:15702. The built-in methods (bevy/query, bevy/get_components, bevy/list+watch, bevy/spawn, …) expose the whole reflected ECS for free.
  2. Custom BRP methods as the game tools: game/state (an optional host-registered snapshot hook — agents work better with a small structured view than raw ECS dumps), game/screenshot + game/screenshot/get (capture via bevy’s Screenshot → PNG on disk → base64 on poll), game/gamepad (mocks bevy_input::gamepad::Gamepad‘s own button/axis state on a synthetic gamepad entity, so injected state flows through input crates’ real binding resolution — dead zones, shared-device ownership, per-context device selection — exactly like a human’s controller), game/keyboard (mocks ButtonInput<KeyCode> directly — see game/keyboard), and game/mouse (mocks ButtonInput<MouseButton> plus real MouseMotion/MouseWheel events, and drives bevy_picking’s own PointerInput pipeline for cursor position and UI clicks — see game/mouse’s doc comment, including a real gotcha found by testing: AccumulatedMouseMotion/AccumulatedMouseScroll can’t be set directly, only injected as events). game/ui is the vision aid that pairs with all of this: an accessibility-tree dump of labeled rects + text in screenshot pixel space, so the model reads rows instead of OCR-ing pixels.
  3. An in-process MCP server (rmcp, Streamable HTTP on 127.0.0.1:15710, stateless mode) whose tools proxy to the BRP methods over loopback HTTP — the MCP layer owns only the protocol surface (tool listing + schemas), never the World (the handlers are async and run outside Bevy’s world; all World access stays in BRP’s systems). read_guide serves the agent guides bundled into the binary at compile time — an agent connected to the MCP server can read the playtesting playbook with zero setup.

Never enable this in player-facing builds: it is a debug/QA tool surface and BRP is unauthenticated by design — localhost bind only.

§Extending with game-specific tools

Two independent extension points, both available to the host app at any time (before or after the plugin is added, in any plugin’s build):

  1. Custom BRP methods — attach a system to the bevy::remote::RemoteMethods resource. Handlers run in the main world with &mut World access, so they can read game state directly:

    fn register_my_methods(app: &mut App) {
        register_game_method(app, "game/my_state", my_game_state);
    }

    Methods can declare a precondition (register_game_method_with_precondition) — a &World (+ intended params) check surfaced by the {prefix}/plan_check pre-flight method and its MCP plan_check tool, so an agent can verify a multi-step sequence would pass before sending it:

    register_game_method_with_precondition(
        &mut app,
        "game/my_state",
        Some(std::sync::Arc::new(in_lobby) as PreconditionFn),
        my_game_state,
    );
  2. Custom MCP tools — pass HarnessTools via McpHarnessConfig::extra_tools. The callback runs on the MCP server thread with parsed arguments and a BrpClient; the conventional shape proxies to a custom BRP method like the one above:

    HarnessTool::new(
        "my_state",
        "Game-specific state snapshot.",
        |client: BrpClient, args: MyToolArgs| async move {
            let mut params = serde_json::json!({});
            if args.detailed {
                params["detailed"] = serde_json::json!(true);
            }
            client.call("game/my_state", params).await
        },
    )

    (The host crate needs serde, schemars, and serde_json in its [dependencies] for the argument struct derives — the harness re-exports BrpClient and HarnessTool itself.)

§Usage

use bevy::prelude::*;
use bevy_mcp_harness::{BevyMcpHarnessPlugin, OffscreenMode};

App::new()
    .add_plugins(BevyMcpHarnessPlugin::default()) // windowed app: tool surfaces only
    // Headless agent host (no window): render into a 1280×800 offscreen texture instead.
    // .add_plugins(BevyMcpHarnessPlugin { config: McpHarnessConfig {
    //     offscreen: OffscreenMode::Owned(UVec2::new(1280, 800)),
    //     ..McpHarnessConfig::from_env() } })
    .run();

Re-exports§

pub use headless::HeadlessUiCameraBootstrap;
pub use headless::NoRenderMode;
pub use headless::OffscreenRenderTarget;
pub use mcp::BrpClient;
pub use mcp::HarnessTool;
pub use schemars;
pub use serde_json;

Modules§

brp
The custom BRP methods — the game tools. Handlers run in the main world with &mut World access (registered as plain systems into bevy::remote::RemoteMethods); the MCP layer proxies to them over loopback HTTP. Ported from PROTOTYPE_19’s dev/tool_api.rs, minus its game-specific methods (game/input action mocks, game/select, game/trigger, game/levels, game/select_level — all bound to that prototype’s own crates).
headless
Headless-mode rendering support: the shared offscreen texture every camera renders into, the camera retargeting that makes window-less apps produce frames, and the render-less (no_render) shim that keeps UI layout working without any render app. Ported from PROTOTYPE_19’s controls/camera.rs.
mcp
The in-process MCP server (rmcp, Streamable HTTP, stateless) — the protocol surface whose tools proxy to the BRP methods in crate::brp over loopback HTTP. The MCP layer owns only the protocol surface (tool listing + schemas), never the World (the handlers are async and run outside Bevy’s world; all World access stays in BRP’s systems).

Structs§

BevyMcpHarnessPlugin
The MCP + Agent playtesting harness. Adds the BRP server (unless the host already did), registers the game/* custom methods, runs the in-process MCP server on a background thread, and — configured for headless use — owns the offscreen render target machinery.
McpHarnessConfig
Harness configuration, inserted as a resource at plugin build. Default is the plain windowed setup (tool surfaces on the default ports, screenshots into <cwd>/mcp_harness/screenshots/); McpHarnessConfig::from_env applies the same CLI flags the prototype used (--brp-port N, --mcp-port N, --no-render).

Enums§

OffscreenMode
The headless rendering mode — see McpHarnessConfig::offscreen.

Constants§

DEFAULT_MCP_PORT
The MCP surface’s default port — NOT 15703, which is bevy_remote’s render-subapp BRP port (DEFAULT_RENDER_PORT, active whenever bevy_render runs): binding the MCP listener there makes the render app’s BRP bind fail and the main BRP pipeline hang in release builds.
DEFAULT_OFFSCREEN_SIZE
The offscreen capture target’s default size (Steam Deck 800p) for McpHarnessConfig users that want the proven headless geometry.

Functions§

isolated_screenshots_dir
Appends the fleet-testing isolation rule to a screenshots base directory: a non-default brp_port captures into a per-client screenshots/client-<port>/ subdirectory instead of the base (a shared directory would cross-contaminate game/screenshot/get polls between concurrently running clients). Hosts that keep their own screenshots convention use this with their base directory so they get the isolation rule for free.
register_game_method
Registers a custom BRP method (the harness’s own game/* methods are registered the same way): system runs in the main world with &mut World access whenever an agent calls <name> over BRP. Must be called after BevyMcpHarnessPlugin (or any other RemotePlugin adder) so the RemoteMethods resource exists — from main, or from a later plugin. Returns the registered system’s id.
register_game_method_with_precondition
register_game_method with a declared PreconditionFn — surfaced by the {prefix}/plan_check pre-flight method (and its MCP plan_check tool) so an agent can check which calls in an intended sequence would fail, and why, before sending them. Advisory only: the method’s own checks remain the source of truth at call time.

Type Aliases§

ClickableFn
The host’s clickable-UI convention for game/ui: return true when entity is a button/interactive node under an interaction convention the harness can’t know (an HTML-markup UI’s click hooks, say). bevy_ui::Interaction holders are always reported clickable; this hook adds to them.
PreconditionFn
A declared precondition for a BRP method, checked by plan_check (pre-flight) before the agent burns a call on a method that would fail. world is the app’s world; params is the params object the agent intends to send (None when the call carries none) — event-shaped methods key their requirements off it. Err(reason) is the human-readable failure the agent would hit.
StateSnapshotFn
The host-registered game/state snapshot: everything an agent needs to reason about this game (app state, the player’s entity/position/health, whatever matters here). Called with &mut World from the BRP handler thread-side systems; return a JSON object. Fused into every game/screenshot/get response (and its .json sidecar) so a screenshot arrives with its ground-truth state attached.