Struct AppState 

#[non_exhaustive]
pub struct AppState {
    pub app: AppInfo,
    pub window: WindowInfo,
    pub elements: Vec<Element>,
    pub truncated: bool,
    pub tree_included: bool,
    pub focused: Option<usize>,
    pub selected_text: Option<String>,
    pub screenshot: Option<Screenshot>,
    pub captured_at: SystemTime,
    pub notes: Vec<String>,
}

A snapshot of one app window.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

app: AppInfo

The app.

window: WindowInfo

The captured window.

elements: Vec<Element>

Accessibility elements in depth-first order; elements[i].index == i.

truncated: bool

Whether node or depth limits cut the tree short.

tree_included: bool

Whether the tree was requested at all.

focused: Option<usize>

Index of the focused element, if it is part of the tree.

selected_text: Option<String>

Currently selected text in the focused element.

screenshot: Option<Screenshot>

Screenshot of the window.

captured_at: SystemTime

When the state was captured.

notes: Vec<String>

Non-fatal problems encountered while capturing, e.g. why the screenshot is missing.

Implementations

impl AppState

pub fn element(&self, index: usize) -> Option<&Element>

Looks up an element by index.

pub fn focused_element(&self) -> Option<&Element>

The focused element.

pub fn find( &self, query: &ElementQuery, ) -> impl Iterator<Item = &Element> + use<'_>

Elements matching query, in tree order.

Examples found in repository

examples/textedit.rs (line 23)

fn main() -> computer_use::Result<()> {
    use std::time::Duration;

    use computer_use::{
        CaptureOptions, ClickMethod, ClickOptions, ClickTarget, Computer, ElementQuery, Error, TextSelection,
    };

    const APP: &str = "TextEdit";
    let mut computer = Computer::new()?.settle_delay(Duration::from_millis(300));
    let tree_only = CaptureOptions::default().include_screenshot(false);

    let state = computer.get_app_state(APP, &tree_only)?;
    let new_doc = state.find(&ElementQuery::text("New Document").with_role("button").exact(true)).next();
    if let Some(button) = new_doc.map(|e| e.index) {
        println!(
            "click New Document: {:?}",
            computer.click(APP, ClickTarget::Element(button), ClickOptions::default())?
        );
    }

    let state = computer.get_app_state(APP, &tree_only)?;
    let text_area = state
        .find(&ElementQuery::role("AXTextArea"))
        .next()
        .map(|e| e.index)
        .ok_or_else(|| Error::StateUnavailable("no text area".into()))?;
    println!(
        "click text area {text_area}: {:?}",
        computer.click(APP, ClickTarget::Element(text_area), ClickOptions::default())?
    );
    computer.set_value(APP, text_area, "")?;
    computer.type_text(APP, "Hello from computer-use-rs!\nSecond line.")?;

    let state = computer.get_app_state(APP, &tree_only)?;
    let value = state.find(&ElementQuery::role("AXTextArea")).next().and_then(|e| e.value.clone());
    println!("text area now contains: {value:?}");

    computer.select_text(APP, text_area, &TextSelection::new("computer-use-rs"))?;
    let state = computer.get_app_state(APP, &tree_only)?;
    println!("selected text: {:?}", state.selected_text);

    computer.press_key(APP, "BackSpace")?;
    let state = computer.get_app_state(APP, &tree_only)?;
    let value = state.find(&ElementQuery::role("AXTextArea")).next().and_then(|e| e.value.clone());
    println!("after BackSpace: {value:?}");

    computer.press_key(APP, "super+a")?;
    let state = computer.get_app_state(APP, &tree_only)?;
    println!("after super+a, selected: {:?}", state.selected_text);

    let focused = ClickOptions::default().method(ClickMethod::FocusedBackground);
    println!("focused click: {:?}", computer.click(APP, ClickTarget::Element(text_area), focused)?);
    computer.type_text(APP, " The end.")?;
    let state = computer.get_app_state(APP, &tree_only)?;
    let value = state.find(&ElementQuery::role("AXTextArea")).next().and_then(|e| e.value.clone());
    println!("final text: {value:?}");

    Ok(())
}

pub fn element_at(&self, point: Point) -> Option<&Element>

The deepest element whose frame contains the screen point, preferring actionable ones.

pub fn render(&self) -> String

Renders the state as compact, human-readable text (also handy as language-model input).

Each element is one tab-indented line beginning with its index:

App=com.apple.TextEdit (pid 812)
Window: "Untitled", App: TextEdit.
0 standard window Untitled
    1 text area (focused, settable, editable) Value: Hello
    2 button Close Secondary Actions: Show Menu

Trait Implementations

impl Clone for AppState

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AppState

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for AppState

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.