bevy-agent-feedback-plugin
Local agent feedback for Bevy apps.
This crate lets Pi/Codex drive a running Bevy app through a v2 JSON-lines TCP protocol. Agents can press keys, move/click/drag/scroll the mouse, submit text/file-drop events, query window metadata, wait for frames, capture PNG screenshots, replay transcripts, and shut the app down cleanly.
Install
[]
= ["dep:bevy-agent-feedback-plugin"]
app.add_plugins;
Install the wrapper binary (bevy-feedback):
Dev form from this repo:
Quick Start
use *;
use ;
use ;
Use port 0 for examples and tests. The plugin writes a self-describing protocol file with the chosen socket, session heartbeat, capture directory, commands, and examples.
For deterministic captures, pin the window size and scale factor:
Window
Drive it
With a separate driver:
The wrapper exports the protocol/capture/artifact paths, waits for readiness, streams logs, releases inputs, sends shutdown, and writes artifacts including a replayable JSONL transcript with responses/timing.
Raw JSON-lines also works:
{"id":1,"command":"window_info"}
{"id":2,"command":"click","x":320,"y":240,"button":"left"}
{"id":3,"command":"key_hold","key":"KeyW","frames":30}
{"id":4,"command":"capture"}
Clients
Rust: bevy_agent_feedback_plugin::client::AgentClient.
Python: clients/python/bevy_feedback.py; skills/driving-bevy-games/drive.py remains a compatibility wrapper.
TypeScript: clients/typescript/bevy_feedback.ts, dependency-free (node with type stripping or tsx).
Rust/Python clients include pixel/OCR assertions. TypeScript v1 covers core driving only. All clients can replay transcript v1 envelopes (request + response + timing) and older request-only JSONL, and release held inputs on close. See skills/driving-bevy-games/SKILL.md for driver workflow details.
import { BevyFeedbackClient } from "./clients/typescript/bevy_feedback.ts";
const game = new BevyFeedbackClient();
console.log(await game.windowInfo());
await game.close();
Optional diagnostics
Enable the diagnostics feature and add AgentFeedbackDiagnosticsPlugin for ecs_summary, list_entities, camera_info, registered state_info, and registered marker-component marker_info commands.
;
app.add_plugins;
CI
See docs/ci-linux.md. Windowed captures need a display (DISPLAY, WAYLAND_DISPLAY, or xvfb-run).