Crate computer_use

Expand description

Cross-platform computer-use primitives: observe desktop apps through their accessibility trees and screenshots, and drive them with accessibility actions and synthetic input.

Use it for UI automation, end-to-end testing of desktop apps, accessibility tooling, or AI agents. It provides:

Computer: a session for observing and acting on apps: list_apps, list_windows, get_app_state, click, perform_secondary_action, scroll, scroll_at, drag, drag_screen, type_text, press_key, set_value, select_text and query. With the native backend it is Send, so it can be shared across threads behind a mutex.
AppState: a captured window with an indexed element tree, a PNG Screenshot, and a compact text rendering for logging, diffing or prompts.
Backend: the OS abstraction. NativeBackend is the implementation for the current platform; implement the trait yourself for tests or other platforms.

§Platform support

Platform	Tree	Screenshot	Input
macOS 14+	Accessibility (`AXUIElement`)	window-server capture, then `ScreenCaptureKit`	`CGEvent` posted to the app’s process (optionally with `SkyLight` synthetic focus), or globally
Windows 10+	UI Automation	`PrintWindow` / GDI	window messages to the app, or `SendInput`
Linux (X11)	AT-SPI over D-Bus	X11 `GetImage`	AT-SPI device event controller (global only)

On macOS the host process needs the Accessibility and Screen Recording permissions; see Computer::permissions and Computer::request_permissions. Without them, apps may appear to have no windows. On Linux, accessibility must be enabled for the session (it is by default on GNOME and KDE); Wayland sessions get trees and actions but no screenshots, and synthetic input only reaches X11/XWayland apps.

§Example

use computer_use::{CaptureOptions, ClickOptions, ClickTarget, Computer, ElementQuery};

let mut computer = Computer::new()?;
let state = computer.get_app_state("TextEdit", &CaptureOptions::default())?;
println!("{state}");

let text_area = state
    .find(&ElementQuery::role("text area"))
    .next()
    .map(|e| e.index)
    .expect("TextEdit has a text area");
computer.click("TextEdit", ClickTarget::Element(text_area), ClickOptions::default())?;
computer.type_text("TextEdit", "Hello from Rust!")?;

Re-exports§

pub use text::SelectionMode;
pub use text::TextSelection;

Modules§

keys: Platform-neutral key chords parsed from xdotool-style specifications.
text: Locating a piece of text inside an element’s value for select_text.

Structs§

Action: An accessibility action an element exposes.
AppInfo: An application that is running or installed.
AppState: A snapshot of one app window.
Capture: What a backend returns from Backend::capture.
CaptureOptions: Limits and toggles for Computer::get_app_state.
CaptureRequest: Parameters for Backend::capture.
ClickOptions: Options for Computer::click.
Computer: A computer-use session over a Backend.
Element: A node in a captured accessibility tree.
ElementData: Raw attributes of an element, as reported by a Backend.
ElementQuery: Criteria for Computer::query and AppState::find.
ElementStates: Boolean accessibility states of an element.
InputTarget: Where synthetic input should be delivered.
MacBackendmacOS: The macOS Backend.
MacElementmacOS: A live macOS accessibility element: the MacBackend handle type.
NativeBackendmacOS and (Linux or macOS or Windows): The macOS Backend.
Permissions: Which OS permissions the current process holds.
Point: A point in logical (non-Retina) screen points, origin at the top-left of the main display.
RawImage: Uncompressed RGBA8 pixels returned by a backend.
RawNode: A raw node returned by a backend, in depth-first pre-order.
Rect: An axis-aligned rectangle in logical points.
Screenshot: A PNG screenshot of a window together with the screen area it shows.
Size: A width and height in logical points.
WindowInfo: The window an AppState was captured from.

Enums§

ClickMethod: How a click is performed.
ClickOutcome: How a click was ultimately delivered.
ClickTarget: What to click.
Delivery: How synthetic input reaches the app.
Error: Everything that can go wrong while observing or driving an app.
MouseButton: A mouse button.
Permission: An OS permission that computer use depends on.
ScrollDirection: A scroll direction.
WindowSelector: Which of an app’s windows to capture.

Traits§

Backend: OS primitives for observing and driving apps.

Type Aliases§

Result: Convenience alias used throughout the crate.