pixelactions 0.2.0

pixelcoords freezes your screen, lets you mark labeled regions, and writes pixel-exact coordinates with crops, drift re-location, and point verification. pixelactions reads that session and acts on it — referencing regions by label, never by raw coordinate, so a run survives the UI moving.

find  →  act  →  assert

No account, no network surface, no daemon — one small native binary that runs, acts, and exits. MIT-licensed, because the aim was to build the best executor in this category and give it away.

Status

Early. The loop works end to end on macOS and on Linux under Wayland (GNOME and KDE): resolve a label to its click point, re-locate it against a fresh capture, act, and confirm. X11 and Windows are next; no crate or binaries are published yet.

Install

Prebuilt binaries will be on the releases page once published — download, unpack, run. Or build it with cargo:

cargo install pixelactions

Rust 1.88+ for the cargo route. pixelactions drives the pixelcoords binary for capture-time work — install both:

cargo install pixelcoords pixelactions

Building on Linux needs the xkbcommon headers, because typing means looking a character up in the compositor's own keymap:

sudo apt-get install -y libxkbcommon-dev pkg-config

The grant, per platform. Either way, pixelactions doctor --probe proves what it can rather than assuming it, and is the right place to answer the prompt — a dialog appearing partway through an unattended run is worse than a refusal.

• macOS asks for an Accessibility grant on first run. It attaches to the terminal that launches pixelactions, not to the binary.
• Linux/Wayland asks you to share a screen, once. The grant is remembered in $XDG_STATE_HOME/pixelactions/, so later runs do not prompt. Sharing is not optional: exact pointer placement is measured against the region the compositor grants with it.

Three ways to drive it

One binary, three surfaces, ranked. Most people want the first. Here is the same task in each — fill a field and confirm the result.

1. Command line

pixelactions run --session ~/captures/checkout \
  click:email type:"a@b.com" key:enter verify:success --yes

Nothing to install, nothing to keep in sync. Verbs chain in one invocation, which also means one relocation pass for the whole sequence.

2. A flow file

session = "~/captures/checkout"

[[step]]
action = "click"
target = "email"

[[step]]
action = "type"
text = "a@b.com"

[[step]]
action = "key"
chord = "enter"

[[step]]
action = "verify"
target = "success"

pixelactions plan --flow checkout.toml       # every coordinate, acts on nothing
pixelactions run  --flow checkout.toml --yes

Same verbs as the command line. Reviewable in a diff — a pull request shows click submit, not arithmetic.

3. The line protocol

ui.send(do="click", target="email")
ui.send(do="type", text=row["email"])
ui.send(do="key", chord="enter")
if ui.send(do="verify", target="success")["outcome"] != "verified":
    failures.append(row)

pixelactions serve --session ~/captures/checkout

One long-lived process speaking JSON on stdin/stdout, so a program in any language owns the loop — branching on what's on screen, retrying with different data, reading a CSV, calling an API between steps. The client above is forty lines of stdlib Python, in docs/PROTOCOL.md.

Escalate on a symptom, not a feature list: one command, then chained commands, then the protocol when you need loops, branching, and data.

There is no embedded interpreter, and never will be. Your bot is written in your language, which is why this works with all of them instead of the two we could afford to embed.

What makes it different

• It acts where regions are now. Before running, every target is re-located against a fresh capture; a region that moved yields corrected coordinates, so a session captured last month still works.
• It refuses rather than guesses. Every region is re-confirmed immediately before the step that touches it, and a region that can't be found unambiguously stops the run with nothing injected. Ambiguity is the test, not distance: a match found in one place is that region however far it moved, which is what lets a flow survive a scrolled page.
• It checks before acting, not after. Acting on something changes it — a focused field grows a caret — so "the region still matches" after a click would mean the click did nothing. Outcomes are asserted by naming what should have changed.
• It distinguishes "executed" from "verified". The OS accepting an event is not the app reacting to one, and the report says which happened.
• Waiting is observable, not hopeful. wait_for polls with real captures and returns the instant the condition holds. No sleeps, at any layer, including the protocol.
• Grabbing the mouse stops it. Slam the cursor into a screen corner and the run halts before the next step — the one control that works while the automation holds your keyboard and the terminal is not focused.
• Exit codes are the API: 0 done, 1 a step failed, 2 malformed question, 3 refused.

Platform status

Verified on Wayland by running the loop: a region marked in pixelcoords on GNOME 46, relocated, clicked, and the application reacted. Placement is exact — a rectangle dragged at (82, 328) recorded as (82, 328).

The Wayland caveat, stated plainly. Wayland exposes no way to ask where the pointer is — the same isolation that makes injection require your consent also hides the pointer from other programs. So the corner kill switch has nothing to watch, and a flow must opt out of it deliberately:

[settings]
failsafe = false

Nothing is faked to avoid this. A stubbed cursor position would either sit in a screen corner and abort every run, or disable the check while appearing to keep it. doctor reports whether your compositor could supply the pointer position through screencast metadata, which is what lifting this needs.

X11 is refused, not approximated: injecting through XWayland reaches X clients only, so the pointer would travel over native windows that never receive the events — a run that clicks through some windows and not others while reporting success. plan works on every session type.

No crate or binaries are published yet. This table is kept honest — claims match runs.

Non-goals

Settled, so the same debates don't reopen. The full list with reasoning is in design/05-NON-GOALS.md.

• No embedded interpreter — not Python, not JS, not Lua. Your bot is written in your language and drives this over a pipe, which is why it works with every language instead of the two we could afford to embed.
• No network surface — no socket, no HTTP, no daemon. This process holds the permission to click and type; a listener would lend that to anything able to reach it.
• No scripting language in flow files — no loops, conditionals, or variables. That's the line between a tool and an RPA suite. Anything needing branching should be a real program calling the protocol.
• No recorder. "Record my clicks and replay them" produces unreviewable artifacts that break on the first UI change. Marks come from pixelcoords, with a human choosing what matters.
• Not an accessibility-tree tool, and not a browser automation tool. Those exist and are good. This is for where trees don't reach.

Documentation

• pixelactions.dev — the website: the loop, comparisons, how-to
• docs/CLI.md — commands, chained verbs, the kill switch, exit codes
• docs/FLOW.md — the flow file: every step and setting
• docs/PROTOCOL.md — the line protocol, with a client in full
• docs/OUTPUT.md — run, plan, and doctor reports
• docs/DEVELOPMENT.md — builds, CI gates, releases
• SKILL.md — for coding agents driving this tool
• design/ — market research, foundations, decisions, milestones, the two-tool contract
• CHANGELOG.md — what changed and why
• CONTRIBUTING.md — bug reports and pull requests

Why this exists

Nothing maintained executes desktop input from declarative files with verification, cross-platform. The near neighbors are Windows-only, macOS-only, mobile-only, or welded to a VM; the incumbent everyone actually uses (PyAutoGUI) is unmaintained with no Wayland support; and computer-use agents shell out to xdotool in containers. Coordinates are the layer that works where accessibility trees don't exist — canvas apps, games, streamed desktops, legacy software.

License

MIT — see LICENSE.

Built by nolindnaidoo.