pixelactions-core

The platform-free core of pixelactions: coordinate-space conversion, the flow-file schema, label resolution, the line protocol's wire types, and the run-report format — with no input synthesis, no OS calls, and #![forbid(unsafe_code)].

Want the tool? Install the binary: cargo install pixelactions. Want to build your own executor, client, or planner? That's this crate.

[dependencies]
pixelactions-core = "0.5"

What you'd use it for

pixelcoords answers where is this thing. pixelactions answers act on it, then confirm it landed. This crate is the half of that with no platform in it — which turns out to be the half that is easy to get wrong.

Build on it when you want to:

• convert a saved coordinate into whatever your input API expects — the single most valuable thing here, and the thing most tools get wrong
• write your own executor over a different input backend (a VM, a remote desktop, a robot arm) while keeping the same flow files
• write a Rust client for the serve line protocol, or an alternative server that speaks it
• parse and validate flow files or chained-argv verbs in your own tooling

The binary is deliberately thin on top of this. Everything it decides about where to act lives here, where it can be unit-tested without a screen.

Coordinate spaces: the reason this crate exists

A pixelcoords session records physical pixels. Input APIs disagree about what they want:

The same saved coordinate therefore needs a different conversion per platform, and getting it wrong does not error — it clicks the wrong place. Space::Auto resolves to whatever the current platform needs, decided in exactly one place (native_space()), so no call site guesses.

use pixelactions_core::convert::{Space, native_space, to_space};
use pixelcoords_core::geometry::{Point, Size};
use pixelcoords_core::session::MonitorRecord;

let monitors = vec![
    // A Retina built-in display at the origin...
    MonitorRecord {
        index: 0,
        name: "Built-in".into(),
        primary: true,
        origin_px: Point::new(0, 0),
        size_px: Size::new(3600, 2338),
        scale: 2.0,
    },
    // ...and a 1x external panel to its left, at negative coordinates.
    MonitorRecord {
        index: 1,
        name: "External".into(),
        primary: false,
        origin_px: Point::new(-3440, 0),
        size_px: Size::new(3440, 1440),
        scale: 1.0,
    },
];

// Physical (850, 440) on the Retina display is logical (425, 220).
let point = to_space(&monitors, 850, 440, Space::Logical).expect("on a monitor");
assert_eq!((point.x, point.y), (425.0, 220.0));
assert_eq!(point.monitor, 0);

// The same physical coordinate on the 1x panel is unchanged, because
// conversion divides by the *containing* monitor's scale — never a
// global one. Mixed-DPI desktops are the normal case, not an edge case.
let external = to_space(&monitors, -2000, 400, Space::Logical).expect("on a monitor");
assert_eq!((external.x, external.y), (-2000.0, 400.0));
assert_eq!(external.monitor, 1);

// A point in the gap between monitors is refused, never clamped —
// clamping would click somewhere plausible and wrong.
assert!(to_space(&monitors, 99_999, 99_999, Space::Logical).is_none());

// What `Auto` means here, decided once:
assert!(matches!(native_space(), Space::Logical | Space::Physical));

monitor_at answers containment on its own, and corners / near_screen_corner back the kill switch — a cursor found in a screen corner stops a run, because grabbing the mouse is what a person does when automation goes wrong.

Flow files

A flow references regions by label, never by raw coordinate. That indirection is the point: a label survives the UI moving, and a diff shows intent ("click submit") rather than arithmetic.

use pixelactions_core::flow::{Flow, Step, Verify};

# fn main() -> Result<(), Box<dyn std::error::Error>> {
let flow = Flow::parse(
    r#"
    session = "~/captures/checkout"

    [settings]
    verify = "each"
    timeout_ms = 30000

    [[step]]
    action = "click"
    target = "email"

    [[step]]
    action = "type"
    text = "a@b.com"

    [[step]]
    action = "wait_for"
    target = "confirmation"
    "#,
)?;

assert_eq!(flow.settings.verify, Verify::Each);
assert_eq!(flow.steps[0], Step::Click { target: "email".into() });

// Every label the flow will touch, for resolving up front.
assert_eq!(flow.targets(), vec!["email", "confirmation"]);
# Ok(())
# }

Parsing is strict: unknown keys are errors, not silent no-ops, so a typo fails at parse time instead of skipping a step at run time. (Session parsing, by contrast, is deliberately tolerant — unknown fields from a newer pixelcoords are ignored, so the two tools release independently.)

Resolving labels to points

plan turns a flow plus a session into concrete coordinates, or refuses. It is total: every label resolves before any action runs, because a half-executed flow is the worst outcome this tool can produce.

use pixelactions_core::convert::Space;
use pixelactions_core::flow::Flow;
use pixelactions_core::plan::{PlanError, plan};
use pixelcoords_core::session::SessionFile;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
let session: SessionFile = serde_json::from_str(EXAMPLE_SESSION)?;
let flow = Flow::parse("session = \"s\"\n\n[[step]]\naction = \"click\"\ntarget = \"submit\"\n")?;

let resolved = plan(&flow, &session, Space::Logical)?;
assert_eq!(resolved.steps[0].points[0].x, 425.0);
assert_eq!(resolved.steps[0].summary, "click submit");

// A missing label fails planning, and the error names what does exist.
let typo = Flow::parse("session = \"s\"\n\n[[step]]\naction = \"click\"\ntarget = \"submti\"\n")?;
let error = plan(&typo, &session, Space::Logical).expect_err("unknown label");
assert!(matches!(error, PlanError::UnknownLabel { .. }));
assert!(error.to_string().contains("submit"));
# Ok(())
# }
#
# const EXAMPLE_SESSION: &str = r#"{
#   "schema": 1,
#   "app": { "name": "pixelcoords", "version": "0.2.1" },
#   "created_utc": "2026-07-29T00:00:00Z",
#   "monitors": [
#     { "index": 0, "name": "Built-in", "primary": true,
#       "origin_px": { "x": 0, "y": 0 },
#       "size_px": { "w": 3600, "h": 2338 }, "scale": 2.0 }
#   ],
#   "selections": [
#     { "shape": "rect", "label": "submit", "monitor": 0,
#       "px":        { "x": 800, "y": 400, "w": 100, "h": 80 },
#       "global_px": { "x": 800, "y": 400, "w": 100, "h": 80 },
#       "crop": "submit.png" }
#   ]
# }"#;

The same verbs, three ways

Chained argv, flow files, and the line protocol all build the same Step, so learning one teaches the others and none can drift.

use pixelactions_core::flow::Step;
use pixelactions_core::protocol::{RequestBody, parse_request};
use pixelactions_core::verb::parse_all;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
// Chained argv: `pixelactions run --session DIR click:submit type:"hi"`
let steps = parse_all(["click:submit", "type:hi", "scroll:results>-3"])?;
assert_eq!(steps[0], Step::Click { target: "submit".into() });

// The line protocol: `do` names the same action a flow file's `action`
// does, so `{"do":"click"}` is the wire form of `action = "click"`.
let request = parse_request(r#"{"id":1,"do":"click","target":"submit"}"#)
    .expect("a valid request");
assert_eq!(request.id, Some(1));
assert_eq!(
    request.body,
    RequestBody::Step { step: Step::Click { target: "submit".into() } }
);
# Ok(())
# }

Parsing a chain is all-or-nothing by design: a typo in step 7 must not perform steps 1 through 6 first.

Speaking the line protocol

protocol holds both directions of the wire format, so you can write a Rust client for pixelactions serve — or an entirely different server that speaks the same thing.

use pixelactions_core::protocol::{PROTOCOL_VERSION, Response, ResponseBody, supported_verbs};

let welcome = Response {
    id: Some(1),
    body: ResponseBody::Welcome {
        version: PROTOCOL_VERSION,
        verbs: supported_verbs(),
        session: "/captures/checkout".into(),
    },
};

// One JSON object per line, newline included — no embedded newlines, in
// either direction.
let line = welcome.to_line();
assert!(line.ends_with('\n'));
assert_eq!(line.matches('\n').count(), 1);
assert!(line.contains("\"result\":\"welcome\""));

The framing rules are the ones LSP, esbuild, and MCP all converged on: one JSON object per line; stdout carries protocol only and stderr is logs; closing stdin is the graceful shutdown; and a version handshake first, so the protocol can change later without breaking programs written against it today.

Reporting what happened

report carries the vocabulary the whole tool reports in — and the distinctions are the point.

use pixelactions_core::report::{RunReport, StepOutcome, StepReport};

// "The OS accepted the event" is not "the app reacted to it".
assert_eq!(StepOutcome::Verified.name(), "verified");
assert_eq!(StepOutcome::Executed.name(), "executed");

// And "it did not work" is not "I declined to try": a refusal is never
// worth retrying, so it earns its own exit code.
let refused = RunReport {
    schema: RunReport::SCHEMA,
    session: "/captures/checkout".into(),
    executed: true,
    steps: vec![StepReport {
        index: 0,
        summary: "click submit".into(),
        outcome: StepOutcome::Refused,
        points: Vec::new(),
        detail: Some("kill switch: the cursor is in a screen corner".into()),
        elapsed_ms: 14,
    }],
};
assert_eq!(refused.exit_code(), 3);

Exit codes are the API: 0 done · 1 a step failed honestly · 2 malformed question · 3 refused.

The modules

Relationship to pixelcoords-core

This crate depends on pixelcoords-core for the session schema, and the dependency is one-way and forever — the two tools release independently and neither is pinned to the other's schedule. Sessions are read through pixelcoords' own types, which ignore unknown fields, so every additive schema change upstream is a no-op here. Our own config is parsed strictly: tolerance is for other people's data, not ours.

Stability, honestly. This is pre-1.0 and shares a version with the binary, so a minor bump can change any signature. The flow file, the line protocol, and the exit codes are the parts with a real compatibility promise — the protocol carries PROTOCOL_VERSION and a handshake precisely so it can change without breaking your program. Pin a caret range and read the CHANGELOG before upgrading.

Every example above is compiled and run as a doctest in CI, so nothing on this page can rot silently.

See also

• pixelcoords — the other half: mark regions, get pixel-exact coordinates
• docs/FLOW.md — every step and setting
• docs/PROTOCOL.md — the line protocol, with a working client
• API docs · repository · pixelactions.dev

MIT licensed.