Trait DesktopBackend 

pub trait DesktopBackend: Send + Sync {
    // Required methods
    fn screenshot(&self, region: Option<Region>) -> Result<Frame, ComputerError>;
    fn move_mouse(&self, x: i32, y: i32) -> Result<(), ComputerError>;
    fn simulate_click(
        &self,
        x: i32,
        y: i32,
        button: MouseButton,
    ) -> Result<(), ComputerError>;
    fn simulate_type(&self, text: &str) -> Result<(), ComputerError>;
    fn simulate_key(
        &self,
        modifiers: &[Modifier],
        key: &str,
    ) -> Result<(), ComputerError>;
    fn simulate_scroll(&self, dx: i32, dy: i32) -> Result<(), ComputerError>;
    fn cursor_position(&self) -> Result<(i32, i32), ComputerError>;
    fn list_displays(&self) -> Result<Vec<DisplayInfo>, ComputerError>;
    fn frontmost(&self) -> FrontmostInfo;
    fn inspect_accessibility(
        &self,
        app_name: Option<&str>,
    ) -> Result<String, ComputerError>;

    // Provided methods
    fn capture_screen(
        &self,
        region: Option<Region>,
    ) -> Result<RawCapture, ComputerError> { ... }
    fn capture_window(
        &self,
        target: &WindowTarget,
    ) -> Result<RawCapture, ComputerError> { ... }
    fn list_apps(&self, _all: bool) -> Result<Vec<AppInfo>, ComputerError> { ... }
    fn list_windows(&self) -> Result<WindowList, ComputerError> { ... }
    fn launch(&self, _app: &str) -> Result<LaunchResult, ComputerError> { ... }
    fn activate(
        &self,
        _app: &str,
        _window: Option<&str>,
    ) -> Result<LaunchResult, ComputerError> { ... }
    fn quit(
        &self,
        _app: &str,
        _force: bool,
    ) -> Result<QuitResult, ComputerError> { ... }
    fn menu_list(
        &self,
        _app: &str,
        _depth: u32,
    ) -> Result<MenuListReport, ComputerError> { ... }
    fn menu_click(
        &self,
        _app: &str,
        _path: &MenuPath,
    ) -> Result<MenuClickReport, ComputerError> { ... }
    fn apple_event(
        &self,
        _app: &str,
        _script: &str,
        _language: AppleEventLanguage,
    ) -> Result<AppleEventReport, ComputerError> { ... }
    fn dock_list(&self) -> Result<DockListReport, ComputerError> { ... }
    fn dock_click(&self, _title: &str) -> Result<DockClickReport, ComputerError> { ... }
    fn spaces(&self) -> Result<SpacesReport, ComputerError> { ... }
    fn doctor(
        &self,
        _options: DoctorOptions,
    ) -> Result<DoctorReport, ComputerError> { ... }
    fn tree(&self, _query: &TreeQuery) -> Result<TreeReport, ComputerError> { ... }
    fn end_session(
        &self,
        _session: Option<&str>,
    ) -> Result<Vec<i32>, ComputerError> { ... }
    fn element_action(
        &self,
        request: &ElementRequest,
    ) -> Result<ElementOutcome, ElementError> { ... }
    fn display_block(&self) -> Option<ComputerError> { ... }
    fn browser_url(&self, pid: i32) -> Result<Option<String>, ComputerError> { ... }
    fn peek_element(
        &self,
        app: Option<&str>,
        window: Option<&str>,
        session: Option<&str>,
        index: u32,
    ) -> Result<Option<ElementPeek>, ComputerError> { ... }
    fn user_input_pause_warning(&self) -> Option<&'static str> { ... }
    fn hit_test(&self, x: i32, y: i32) -> Result<Option<HitInfo>, ComputerError> { ... }
    fn input_support(&self) -> Capabilities { ... }
    fn marks_for_tree(
        &self,
        _query: &TreeQuery,
        _map: &ScreenshotMap,
    ) -> Result<Vec<MarkBox>, ComputerError> { ... }
    fn recognize_text(
        &self,
        _image: &RgbaImage,
        _map: &ScreenshotMap,
    ) -> Result<OcrResult, ComputerError> { ... }
    fn secure_ocr_frames(
        &self,
        _query: &TreeQuery,
        _map: &ScreenshotMap,
    ) -> Result<Vec<ImageRect>, ComputerError> { ... }
    fn clipboard_get(&self) -> Result<ClipboardContents, ComputerError> { ... }
    fn clipboard_set_text(&self, text: &str) -> Result<(), ComputerError> { ... }
    fn clipboard_set_image(
        &self,
        width: u32,
        height: u32,
        rgba: &[u8],
    ) -> Result<(), ComputerError> { ... }
    fn release_held(&self) -> Result<(), ComputerError> { ... }
    fn perform_input(
        &self,
        job: &InputJob,
    ) -> Result<InputReport, ComputerError> { ... }
}

Desktop automation surface used by crate::dispatch.

Also exported as Backend.

Required Methods

fn screenshot(&self, region: Option<Region>) -> Result<Frame, ComputerError>

Capture the desktop, or a logical region of it, as RGBA pixels.

fn move_mouse(&self, x: i32, y: i32) -> Result<(), ComputerError>

Move the pointer without clicking.

fn simulate_click( &self, x: i32, y: i32, button: MouseButton, ) -> Result<(), ComputerError>

Move the pointer to (x, y) and click once.

fn simulate_type(&self, text: &str) -> Result<(), ComputerError>

Type Unicode text into the focused control.

fn simulate_key( &self, modifiers: &[Modifier], key: &str, ) -> Result<(), ComputerError>

Press modifiers, tap key, then release the modifiers.

key is a canonical name from crate::keys::canonical_key.

fn simulate_scroll(&self, dx: i32, dy: i32) -> Result<(), ComputerError>

Scroll by wheel steps. Positive dy scrolls down. Positive dx scrolls right.

fn cursor_position(&self) -> Result<(i32, i32), ComputerError>

Current pointer position in logical pixels.

fn list_displays(&self) -> Result<Vec<DisplayInfo>, ComputerError>

Connected displays. Does not capture pixels.

fn frontmost(&self) -> FrontmostInfo

Frontmost application and window. Never panics; missing data is None.

fn inspect_accessibility( &self, app_name: Option<&str>, ) -> Result<String, ComputerError>

Accessibility tree as a JSON string, when the platform can produce one.

Provided Methods

fn capture_screen( &self, region: Option<Region>, ) -> Result<RawCapture, ComputerError>

Desktop or region capture plus the map inputs (origin and logical size).

fn capture_window( &self, target: &WindowTarget, ) -> Result<RawCapture, ComputerError>

Capture one window. macOS uses ScreenCaptureKit. Other platforms crop the desktop image when the bounds are known.

fn list_apps(&self, _all: bool) -> Result<Vec<AppInfo>, ComputerError>

Running applications. all includes accessory and prohibited apps.

fn list_windows(&self) -> Result<WindowList, ComputerError>

On-screen windows in z-order. Index 0 is front.

fn launch(&self, _app: &str) -> Result<LaunchResult, ComputerError>

Launch an application by name, bundle id, or path.

fn activate( &self, _app: &str, _window: Option<&str>, ) -> Result<LaunchResult, ComputerError>

Bring an application forward. window selects a window inside it.

fn quit(&self, _app: &str, _force: bool) -> Result<QuitResult, ComputerError>

Ask an application to quit. force uses a forced terminate where the OS has one.

fn menu_list( &self, _app: &str, _depth: u32, ) -> Result<MenuListReport, ComputerError>

Walk app’s menu bar. The macOS backend uses AXMenuBar -> AXMenuBarItem -> AXMenu. Other platforms return unsupported. depth caps recursion; out-of-depth submenus are reported as a single placeholder child.

fn menu_click( &self, _app: &str, _path: &MenuPath, ) -> Result<MenuClickReport, ComputerError>

Press one menu item by ">"-separated titles. A miss is element_not_found. A disabled item is execution_failed.

fn apple_event( &self, _app: &str, _script: &str, _language: AppleEventLanguage, ) -> Result<AppleEventReport, ComputerError>

Run an Apple event via osascript. macOS only. script is the body passed with -e; it never goes through a shell. The script body is never logged verbatim; only its byte count and SHA-256 are recorded.

fn dock_list(&self) -> Result<DockListReport, ComputerError>

List Dock items. macOS only.

fn dock_click(&self, _title: &str) -> Result<DockClickReport, ComputerError>

Click a Dock item by title. macOS only. A miss is element_not_found.

fn spaces(&self) -> Result<SpacesReport, ComputerError>

Group on-screen windows by the screen that contains each window’s center. Mission Control moves are not attempted; the skylight feature, when active, may opt into best-effort space moves.

fn doctor(&self, _options: DoctorOptions) -> Result<DoctorReport, ComputerError>

Permission probe. options.prompt may show a system prompt on macOS.

The default does not prompt and does not open Settings.

fn tree(&self, _query: &TreeQuery) -> Result<TreeReport, ComputerError>

Numbered accessibility tree for the key window.

fn end_session(&self, _session: Option<&str>) -> Result<Vec<i32>, ComputerError>

Restore accessibility flags recorded for session.

The returned pids are the processes whose flags were written back. An empty list means there was nothing to restore.

fn element_action( &self, request: &ElementRequest, ) -> Result<ElementOutcome, ElementError>

Press, focus, set a value, select text, perform an action, scroll, or type into one cached element. The element is revalidated before the action.

fn display_block(&self) -> Option<ComputerError>

Locked, asleep, or no display. None when capture may proceed.

The default is None so the stub does not read the live session. The macOS backend overrides it.

fn browser_url(&self, pid: i32) -> Result<Option<String>, ComputerError>

Page URL for a browser process. Ok(None) when the platform has no URL.

A ComputerError::PermissionRequired fails closed when a block list caused the call. Other platforms return Ok(None).

fn peek_element( &self, app: Option<&str>, window: Option<&str>, session: Option<&str>, index: u32, ) -> Result<Option<ElementPeek>, ComputerError>

Title and secure bit of a cached element. Ok(None) when it is not cached.

The default on macOS reads the session cache. The stub reads its fixture.

fn user_input_pause_warning(&self) -> Option<&'static str>

Install the listen-only user-input tap when this process should pause.

None when the tap is installed or was not requested. The warning string is user-input pause unavailable. Tests never create a real event tap.

fn hit_test(&self, x: i32, y: i32) -> Result<Option<HitInfo>, ComputerError>

Hit-test a screen point. macOS reports the element under the point. Other platforms return Ok(None) and the coordinate click still runs.

fn input_support(&self) -> Capabilities

Whether background event posting exists. The stub overrides this.

fn marks_for_tree( &self, _query: &TreeQuery, _map: &ScreenshotMap, ) -> Result<Vec<MarkBox>, ComputerError>

Walk the latest tree and return rectangles in image pixels plus the label to draw (n for tree index). The screenshot command uses these for the --marks overlay.

Empty results are not an error; the dispatcher decides whether to fall back to OCR or to leave the image unmarked. Non-macOS platforms always return Ok(Vec::new()); OCR for those platforms is Unsupported.

fn recognize_text( &self, _image: &RgbaImage, _map: &ScreenshotMap, ) -> Result<OcrResult, ComputerError>

Run OCR on one captured image and return the redacted, screen-remapped blocks. image is the prepared shot’s pixels.

fn secure_ocr_frames( &self, _query: &TreeQuery, _map: &ScreenshotMap, ) -> Result<Vec<ImageRect>, ComputerError>

Image-pixel rectangles of every secure element in the latest tree. The OCR dispatcher drops recognized text whose block falls inside any of these rectangles. The default returns an empty list, which matches every non-macOS platform and the mock backend.

map is the screenshot map of the capture the OCR ran on. Platform frames are screen points, so redaction is only correct when the backend converts through it — P12b wires that up on Windows and finishes the macOS placeholder it replaces.

fn clipboard_get(&self) -> Result<ClipboardContents, ComputerError>

Read the clipboard. The default uses the system pasteboard outside tests.

The stub overrides this and never touches the real pasteboard.

fn clipboard_set_text(&self, text: &str) -> Result<(), ComputerError>

Write clipboard text. Empty text is refused before this is called.

fn clipboard_set_image( &self, width: u32, height: u32, rgba: &[u8], ) -> Result<(), ComputerError>

Write an RGBA image to the clipboard.

fn release_held(&self) -> Result<(), ComputerError>

Release mouse buttons and keys this process still holds.

The real registry is process-global. The stub overrides this method.

fn perform_input(&self, job: &InputJob) -> Result<InputReport, ComputerError>

Run one input tier. The default posts nothing.

event and synthetic are unsupported off macOS. activate uses the existing foreground enigo path and does not restore focus.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl DesktopBackend for LinuxDesktopBackend