Crate computer_use 

Cross-platform desktop automation for agents.

The CLI and the library share dispatch: a typed Action goes in and a Response comes out. Platform backends live behind DesktopBackend.

Example

use computer_use::{dispatch, Action};

let response = dispatch(Action::Wait {
    ms: Some(0),
    seconds: None,
});
assert!(response.ok);
assert_eq!(response.to_value()["waited_ms"].as_u64(), Some(0));

Re-exports

pub use backend::create_desktop_backend;

pub use backend::Backend;

pub use backend::DesktopBackend;

pub use error::ComputerError;

pub use error::ErrorBody;

pub use keys::parse_key_combo;

pub use keys::parse_key_sequence;

pub use keys::KeyCombo;

pub use protocol::dispatch;

pub use protocol::dispatch_context;

pub use protocol::dispatch_with;

pub use protocol::handle_request_line;

pub use protocol::handle_request_line_with;

pub use protocol::parse_request;

pub use protocol::Action;

pub use protocol::CallContext;

pub use protocol::Request;

pub use protocol::Response;

pub use types::parse_path;

pub use types::parse_region;

pub use types::DisplayInfo;

pub use types::Frame;

pub use types::FrontmostInfo;

pub use types::ImageFormat;

pub use types::Modifier;

pub use types::MouseButton;

pub use types::Point;

pub use types::Region;

pub use types::ScrollDirection;

pub use backend::LinuxDesktopBackend;

Modules

adapters

Schema translators for exec and serve.

audit

Append-only JSONL audit log.

backend

Platform desktop backends.

browser

P10: Chrome DevTools Protocol (CDP) attach-only client.

cache

Per-session tree snapshots.

element

Element actions: revalidation, the perform allowlist, and secure-field refusal.

error

Error type shared by the library and the CLI JSON envelope.

isolation

P14 — display filtering and the honest isolation matrix.

keys

Parse key combinations such as cmd+shift+4.

menu_path

Parse "File > Save"-style menu paths into a typed list of titles.

overlay

Agent-cursor overlay protocol.

policy

Application policy: built-in deny list plus user and admin policy.toml.

protocol

Actions, JSON requests, and the shared dispatcher.

scroll

Scroll verification and the AX scroll-bar fallback. Pure: no events are posted, no Accessibility call is made, and no clock is read from here.

shortcuts

Run a Shortcuts workflow by name through the shortcuts binary.

state

Best-effort desktop snapshot. Subsystem failures become null fields and a note.

target

Application and window selectors.

tree

Numbered accessibility tree text, diff, and the pure helpers the AX walker uses.

types

Shared geometry, input, and image types.

vision

Vision utilities: pure-image marks overlay and the OCR backend selector.

window_groups

Group on-screen windows by the screen that contains each window’s center.

Structs

EncodedImage

Encoded screenshot bytes and the final pixel size.

InputSession

Guard that releases held buttons and keys when the last one drops.

MapFormulas

The two formulas from the screenshot-map contract.

MapPoint

Integer point stored on the screenshot map.

ScreenshotMap

Map attached to every screenshot and stored in the session cache.

Functions

encode_image

Resize to max_width when the frame is wider, then encode.

image_to_screen

Image pixel to screen point.

screen_to_image

Screen point to image pixel.

shutdown_sessions

Restore Electron accessibility flags and stop AX observers.