pixelactions 0.1.0

Execute desktop interactions from pixelcoords sessions: resolve a labeled region, act at the verified point, confirm it landed
pixelactions-0.1.0 is not a library.

pixelcoords freezes your screen, lets you mark labeled regions, and writes pixel-exact coordinates with crops, drift re-location, and point verification. pixelactions reads that session and acts on it — referencing regions by label, never by raw coordinate, so a run survives the UI moving.

find  →  act  →  assert

No account, no network surface, no daemon — one small native binary that runs, acts, and exits. MIT-licensed, because the aim was to build the best executor in this category and give it away.

Status

Early, and macOS only. The loop works end to end: resolve a label to its click point, re-locate it against a fresh capture, act, and confirm. Windows and X11 are next; no crate or binaries are published yet.

Install

Prebuilt binaries will be on the releases page once published — download, unpack, run. Or build it with cargo:

cargo install pixelactions

Rust 1.88+ for the cargo route. pixelactions drives the pixelcoords binary for capture-time work — install both:

cargo install pixelcoords pixelactions

macOS asks for an Accessibility grant on first run. The grant attaches to the terminal that launches pixelactions, not the binary — pixelactions doctor --probe proves the grant instead of assuming it.

Three ways to drive it

One binary, three surfaces, ranked. Most people want the first. Here is the same task in each — fill a field and confirm the result.

1. Command line

pixelactions run --session ~/captures/checkout \
  click:email type:"a@b.com" key:enter verify:success --yes

Nothing to install, nothing to keep in sync. Verbs chain in one invocation, which also means one relocation pass for the whole sequence.

2. A flow file

session = "~/captures/checkout"

[[step]]
action = "click"
target = "email"

[[step]]
action = "type"
text = "a@b.com"

[[step]]
action = "key"
chord = "enter"

[[step]]
action = "verify"
target = "success"

pixelactions plan --flow checkout.toml       # every coordinate, acts on nothing
pixelactions run  --flow checkout.toml --yes

Same verbs as the command line. Reviewable in a diff — a pull request shows click submit, not arithmetic.

3. The line protocol

ui.send(do="click", target="email")
ui.send(do="type", text=row["email"])
ui.send(do="key", chord="enter")
if ui.send(do="verify", target="success")["outcome"] != "verified":
    failures.append(row)

pixelactions serve --session ~/captures/checkout

One long-lived process speaking JSON on stdin/stdout, so a program in any language owns the loop — branching on what's on screen, retrying with different data, reading a CSV, calling an API between steps. The client above is forty lines of stdlib Python, in docs/PROTOCOL.md.

Escalate on a symptom, not a feature list: one command, then chained commands, then the protocol when you need loops, branching, and data.

There is no embedded interpreter, and never will be. Your bot is written in your language, which is why this works with all of them instead of the two we could afford to embed.

What makes it different

  • It acts where regions are now. Before running, every target is re-located against a fresh capture; a region that moved yields corrected coordinates, so a session captured last month still works.
  • It refuses rather than guesses. Every region is re-confirmed immediately before the step that touches it, and a region that can't be found unambiguously stops the run with nothing injected. Ambiguity is the test, not distance: a match found in one place is that region however far it moved, which is what lets a flow survive a scrolled page.
  • It checks before acting, not after. Acting on something changes it — a focused field grows a caret — so "the region still matches" after a click would mean the click did nothing. Outcomes are asserted by naming what should have changed.
  • It distinguishes "executed" from "verified". The OS accepting an event is not the app reacting to one, and the report says which happened.
  • Waiting is observable, not hopeful. wait_for polls with real captures and returns the instant the condition holds. No sleeps, at any layer, including the protocol.
  • Grabbing the mouse stops it. Slam the cursor into a screen corner and the run halts before the next step — the one control that works while the automation holds your keyboard and the terminal is not focused.
  • Exit codes are the API: 0 done, 1 a step failed, 2 malformed question, 3 refused.

Platform status

Platform State
macOS Supported — the loop works end to end; primary development platform
Windows Next — the goal is the same flow file running unmodified
Linux (X11) Next — alongside Windows
Linux (Wayland) Later — after Windows and X11

No crate or binaries are published yet. This table is kept honest — claims match runs.

Non-goals

Settled, so the same debates don't reopen. The full list with reasoning is in design/05-NON-GOALS.md.

  • No embedded interpreter — not Python, not JS, not Lua. Your bot is written in your language and drives this over a pipe, which is why it works with every language instead of the two we could afford to embed.
  • No network surface — no socket, no HTTP, no daemon. This process holds the permission to click and type; a listener would lend that to anything able to reach it.
  • No scripting language in flow files — no loops, conditionals, or variables. That's the line between a tool and an RPA suite. Anything needing branching should be a real program calling the protocol.
  • No recorder. "Record my clicks and replay them" produces unreviewable artifacts that break on the first UI change. Marks come from pixelcoords, with a human choosing what matters.
  • Not an accessibility-tree tool, and not a browser automation tool. Those exist and are good. This is for where trees don't reach.

Documentation

Why this exists

Nothing maintained executes desktop input from declarative files with verification, cross-platform. The near neighbors are Windows-only, macOS-only, mobile-only, or welded to a VM; the incumbent everyone actually uses (PyAutoGUI) is unmaintained with no Wayland support; and computer-use agents shell out to xdotool in containers. Coordinates are the layer that works where accessibility trees don't exist — canvas apps, games, streamed desktops, legacy software.

License

MIT — see LICENSE.

Built by nolindnaidoo.