pub enum PlannedAction {
Show 29 variants
Click {
target_id: String,
expect_after: Option<EffectExpectation>,
},
Type {
target_id: Option<String>,
text: String,
},
Key {
key: String,
},
KeyCombo {
keys: Vec<String>,
},
SetValue {
target_id: String,
value: String,
expect_after: Option<EffectExpectation>,
},
Scroll {
dx: i32,
dy: i32,
},
Drag {
from_target_id: String,
to_target_id: String,
},
Wait {
ms: u32,
},
Custom {
adapter: String,
action: String,
params: Value,
},
Extract {
goal: String,
data: String,
},
Batch {
actions: Vec<PlannedAction>,
},
Act {
instruction: String,
},
Done {
summary: String,
evidence_ids: Vec<String>,
},
Fail {
reason: String,
},
AxAction {
target_id: String,
action: String,
label: Option<String>,
role_hint: Option<String>,
expect_after: Option<EffectExpectation>,
},
ActivateApp {
app_name: String,
},
LaunchApp {
app_name: String,
background: bool,
},
QuitApp {
app_name: String,
},
Select {
from_x: i32,
from_y: i32,
to_x: i32,
to_y: i32,
},
CdpEval {
expression: String,
},
Navigate {
url: String,
wait_until: Option<String>,
timeout_ms: Option<u64>,
dismiss_overlays: Option<bool>,
},
NotebookWrites {
key: String,
value: String,
category: String,
},
ExtractWithFallback {
name: String,
selectors: Vec<String>,
parse_as: String,
},
WriteCells {
app: String,
sheet: Option<String>,
table: Option<String>,
writes: Vec<CellWrite>,
verify: bool,
},
ReadCells {
app: String,
sheet: Option<String>,
table: Option<String>,
cell_refs: Vec<String>,
},
Window {
op: String,
app: Option<String>,
window_index: usize,
x: Option<f64>,
y: Option<f64>,
width: Option<f64>,
height: Option<f64>,
preset: Option<String>,
display: Option<usize>,
},
Dialog {
op: String,
button: Option<String>,
value: Option<String>,
field_index: usize,
},
Dock {
op: String,
name: Option<String>,
},
MenuExtra {
op: String,
name: Option<String>,
},
}Expand description
The action the planner wants to execute.
Variants§
Click
Fields
expect_after: Option<EffectExpectation>Optional post-state verification. When set, the runtime
polls the page after the click and reports the action as
failed if the expectation doesn’t hold within timeout_ms.
Closes the “click reported ok but DOM didn’t react” gap.
Type
Fields
Key
KeyCombo
SetValue
Set a value directly via the accessibility API (bypasses mouse/keyboard). More reliable than Type for form fields where settable=true.
Fields
expect_after: Option<EffectExpectation>Optional post-state verification — see EffectExpectation.
Most useful for <select> where setting the value should
flip an aria-selected attribute on the chosen option, or
for <input type="text"> in framework-controlled forms
where the visible label should reflect the typed value.
Scroll
Drag
Drag from one element/position to another.
Wait
Custom
Extract
Extract: read data from the current page/screen without interaction.
For read-only goals: “what are the prices?”, “read the headlines”, “what’s on screen?”
Returns the extracted data in the data field — no clicking or navigation needed.
Fields
Batch
Batch: execute multiple simple actions in sequence without re-planning.
Fields
actions: Vec<PlannedAction>Act
Natural language action — a runtime resolves the instruction to the best matching element.
Done
Terminal: goal achieved. evidence_ids should cite element IDs
from the current context that prove the goal was achieved.
Fail
Terminal: cannot proceed.
AxAction
Native accessibility action — more reliable than coordinate clicks for desktop apps. Uses macOS AXUIElementPerformAction under the hood.
Fields
target_id: StringAX hash id from the same perception snapshot that produced
the plan. May be missing (empty / null) when the planner
only knows the visible label — the runtime can fall back to
label + role_hint resolution in that case.
Lenient deserialization accepts null from the LLM (some
models emit "target_id": null when they want label-only
dispatch) and treats it as an empty string. Without this,
the whole turn parse-errors with
invalid type: null, expected a string and the run dies
before the runner gets a chance to fall back.
action: StringThe action to perform: “click” (AXPress), “activate” (AXConfirm), “increment”, “decrement”, “show_menu”
label: Option<String>Label hint for fallback element resolution. AX IDs are hashes
that include bounds + depth and therefore change whenever the
UI mutates between plan time and dispatch time. If the runtime
can’t find target_id in the live AX tree (or target_id
is missing entirely), it falls back to searching for the
first visible element whose role matches role_hint (if
provided) and whose label equals label. Planner must
populate this from the same perception snapshot that
produced the target_id.
expect_after: Option<EffectExpectation>Optional post-state verification — see EffectExpectation.
Same shape as the equivalent field on Click/SetValue.
Mostly useful when the AX action lands on a browser DOM
element (the runtime routes those through CDP), so the page
state is observable via querySelector.
ActivateApp
Activate (bring to front) a macOS application by name.
Uses open -a under the hood — the most reliable app switching method.
LaunchApp
Launch (start) a macOS application by name. Unlike activate_app, this
is about starting the app; with background it launches without
stealing focus (open -g). Verified by polling until the process
appears.
Fields
QuitApp
Quit a macOS application by name, gracefully (AppleScript quit, like
⌘Q). Never force-kills — an app showing an unsaved-changes dialog stays
running and the action reports failure. Verified by polling until the
process is gone.
Select
Select text by dragging from one coordinate to another. Used for text selection, highlighting, and marking tasks.
CdpEval
Execute JavaScript in the focused browser tab via Chrome DevTools Protocol. Fastest way to interact with web pages: click elements, fill forms, extract data, dismiss cookie banners — all in a single action.
Canonical navigation action. A runtime may route this through a browser
adapter, CDP, WebDriver, or another navigation backend and then wait
according to wait_until.
All three control fields are optional with sensible defaults so
the legacy {"type":"navigate","url":"..."} payload still parses
unchanged. Aliases href / to on url survive too — LLMs
gravitate toward both.
NotebookWrites
No-op: LLM sometimes puts notebook_writes inside the actions array. This variant absorbs that mistake gracefully instead of causing a parse error. The actual notebook_writes are processed from the top-level PlannedStep field.
ExtractWithFallback
Declarative extraction with selector fallbacks.
Replaces the “LLM hand-writes document.querySelector(...) in a
loop” failure mode. Runtime tries each selector in order via
CDP Runtime.evaluate, parses according to parse_as, and on
first success writes the value into shared_memory[name]. The
planner supplies the scenario-specific selector knowledge as
parameters; the retry/parse machinery is generic.
Contract with the runner: consecutive failures for the same
name (across turns) accumulate toward an auto-null cutoff
(see the stall/retry budget in canonical_runner.rs). After
the cutoff the runtime records shared_memory[name] = null so
the LLM stops polishing a field that the page does not surface.
Fields
name: StringLogical name for this extraction target (e.g. "btc_price").
Written into shared_memory on success, and used by the
runner to group consecutive failures for retry budgeting.
WriteCells
Deterministic spreadsheet cell writes via AppleScript (Numbers).
Replaces the flaky keystroke recipe (activate_app → key(arrows) → key(Delete) → type → key(Return)) with one atomic operation
against the document model. The keystroke recipe produced
concatenated garbage values, duplicated headers, and values
landing in the wrong cells whenever an intermediate step got
perturbed by focus drift or AX tree lag. WriteCells sidesteps
the entire UI event loop.
Batch-shaped because the AppleScript spawn cost amortizes across
many cells — single-cell callers pass a length-1 writes vector.
Fields
ReadCells
Deterministic spreadsheet cell reads via AppleScript (Numbers).
Use this when the agent needs spreadsheet truth from the app model instead of relying on the accessibility tree to expose the values. Particularly useful for Numbers, whose AX surface often only exposes the focused cell or formula-bar content.
Fields
Window
Native window management — move/resize/minimize/maximize/focus a macOS
window resolved by app + window_index. A runtime executes the
operation and reads geometry back into the receipt. op is one of:
"move", "resize", "set_bounds",
"minimize", "unminimize", "maximize", "focus". A preset
(WS2.3 tiling, e.g. "left_half") overrides op + geometry when set.
Fields
Dialog
Native dialog / sheet driver (WS5) — list, click a button by title, set
a text field, or dismiss the frontmost macOS Open/Save/Print sheet or
alert. The runtime resolves the dialog’s controls via the accessibility
tree. op is one of: "list", "click", "set_field", "dismiss".
Fields
Button title to click (for op = "click"); case-insensitive substring.
Dock
Native Dock control (WS6) — list items, launch / right-click an item by
title, or toggle auto-hide. op is one of: "list", "launch",
"right_click", "hide", "show".
MenuExtra
Native menu-bar extras (WS7) — list system status items (Wi-Fi,
Bluetooth, Control Center, …) or click one by title. op is "list"
or "click".
Implementations§
Source§impl PlannedAction
impl PlannedAction
Sourcepub fn target_ids(&self) -> Vec<&str>
pub fn target_ids(&self) -> Vec<&str>
Target element IDs this action depends on, if any. Used by the runner to pre-validate that planned elements still exist in the fresh runtime context before dispatch — missing targets trigger a replan instead of silent misfire.
Returns an empty slice for actions with no element target (coordinate
scrolls, key events, CdpEval, terminal actions). Drag returns two
IDs (from + to). Batch recurses — every sub-action’s targets are
collected flat, so if any one is missing the whole batch is aborted
before any side-effect lands. Sub-batches inside batches collapse the
same way.
Trait Implementations§
Source§impl Clone for PlannedAction
impl Clone for PlannedAction
Source§fn clone(&self) -> PlannedAction
fn clone(&self) -> PlannedAction
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more