Crate pixelactions_core 

Platform-free core for pixelactions.

Everything here is arithmetic and data: parsing a flow file, resolving its steps against a pixelcoords session, and converting a physical pixel into the coordinate space a platform’s input API expects. No window system, no OS calls, no capture — those live in the binary.

The split exists for the same reason it does in pixelcoords: the part that decides where to click must be testable without a screen.

The crate README is included below, which makes every example on the crates.io page a compiled doctest — documentation that cannot rot without failing CI.

pixelactions-core

The platform-free core of pixelactions: coordinate-space conversion, the flow-file schema, label resolution, the line protocol’s wire types, and the run-report format — with no input synthesis, no OS calls, and #![forbid(unsafe_code)].

Want the tool? Install the binary: cargo install pixelactions. Want to build your own executor, client, or planner? That’s this crate.

[dependencies]
pixelactions-core = "0.4"

What you’d use it for

pixelcoords answers where is this thing. pixelactions answers act on it, then confirm it landed. This crate is the half of that with no platform in it — which turns out to be the half that is easy to get wrong.

Build on it when you want to:

• convert a saved coordinate into whatever your input API expects — the single most valuable thing here, and the thing most tools get wrong
• write your own executor over a different input backend (a VM, a remote desktop, a robot arm) while keeping the same flow files
• write a Rust client for the serve line protocol, or an alternative server that speaks it
• parse and validate flow files or chained-argv verbs in your own tooling

The binary is deliberately thin on top of this. Everything it decides about where to act lives here, where it can be unit-tested without a screen.

Coordinate spaces: the reason this crate exists

A pixelcoords session records physical pixels. Input APIs disagree about what they want:

Platform	Input API	Speaks
macOS	CGEvent	logical points, global space, origin top-left
Windows	SendInput	physical pixels, normalized across the virtual desktop
Linux / X11	XTEST	physical pixels on the root window

The same saved coordinate therefore needs a different conversion per platform, and getting it wrong does not error — it clicks the wrong place. Space::Auto resolves to whatever the current platform needs, decided in exactly one place (native_space()), so no call site guesses.

use pixelactions_core::convert::{Space, native_space, to_space};
use pixelcoords_core::geometry::{Point, Size};
use pixelcoords_core::session::MonitorRecord;

let monitors = vec![
    // A Retina built-in display at the origin...
    MonitorRecord {
        index: 0,
        name: "Built-in".into(),
        primary: true,
        origin_px: Point::new(0, 0),
        size_px: Size::new(3600, 2338),
        scale: 2.0,
    },
    // ...and a 1x external panel to its left, at negative coordinates.
    MonitorRecord {
        index: 1,
        name: "External".into(),
        primary: false,
        origin_px: Point::new(-3440, 0),
        size_px: Size::new(3440, 1440),
        scale: 1.0,
    },
];

// Physical (850, 440) on the Retina display is logical (425, 220).
let point = to_space(&monitors, 850, 440, Space::Logical).expect("on a monitor");
assert_eq!((point.x, point.y), (425.0, 220.0));
assert_eq!(point.monitor, 0);

// The same physical coordinate on the 1x panel is unchanged, because
// conversion divides by the *containing* monitor's scale — never a
// global one. Mixed-DPI desktops are the normal case, not an edge case.
let external = to_space(&monitors, -2000, 400, Space::Logical).expect("on a monitor");
assert_eq!((external.x, external.y), (-2000.0, 400.0));
assert_eq!(external.monitor, 1);

// A point in the gap between monitors is refused, never clamped —
// clamping would click somewhere plausible and wrong.
assert!(to_space(&monitors, 99_999, 99_999, Space::Logical).is_none());

// What `Auto` means here, decided once:
assert!(matches!(native_space(), Space::Logical | Space::Physical));

monitor_at answers containment on its own, and corners / near_screen_corner back the kill switch — a cursor found in a screen corner stops a run, because grabbing the mouse is what a person does when automation goes wrong.

Flow files

A flow references regions by label, never by raw coordinate. That indirection is the point: a label survives the UI moving, and a diff shows intent (“click submit”) rather than arithmetic.

use pixelactions_core::flow::{Flow, Step, Verify};

let flow = Flow::parse(
    r#"
    session = "~/captures/checkout"

    [settings]
    verify = "each"
    timeout_ms = 30000

    [[step]]
    action = "click"
    target = "email"

    [[step]]
    action = "type"
    text = "a@b.com"

    [[step]]
    action = "wait_for"
    target = "confirmation"
    "#,
)?;

assert_eq!(flow.settings.verify, Verify::Each);
assert_eq!(flow.steps[0], Step::Click { target: "email".into() });

// Every label the flow will touch, for resolving up front.
assert_eq!(flow.targets(), vec!["email", "confirmation"]);

Parsing is strict: unknown keys are errors, not silent no-ops, so a typo fails at parse time instead of skipping a step at run time. (Session parsing, by contrast, is deliberately tolerant — unknown fields from a newer pixelcoords are ignored, so the two tools release independently.)

Resolving labels to points

plan turns a flow plus a session into concrete coordinates, or refuses. It is total: every label resolves before any action runs, because a half-executed flow is the worst outcome this tool can produce.

use pixelactions_core::convert::Space;
use pixelactions_core::flow::Flow;
use pixelactions_core::plan::{PlanError, plan};
use pixelcoords_core::session::SessionFile;

let session: SessionFile = serde_json::from_str(EXAMPLE_SESSION)?;
let flow = Flow::parse("session = \"s\"\n\n[[step]]\naction = \"click\"\ntarget = \"submit\"\n")?;

let resolved = plan(&flow, &session, Space::Logical)?;
assert_eq!(resolved.steps[0].points[0].x, 425.0);
assert_eq!(resolved.steps[0].summary, "click submit");

// A missing label fails planning, and the error names what does exist.
let typo = Flow::parse("session = \"s\"\n\n[[step]]\naction = \"click\"\ntarget = \"submti\"\n")?;
let error = plan(&typo, &session, Space::Logical).expect_err("unknown label");
assert!(matches!(error, PlanError::UnknownLabel { .. }));
assert!(error.to_string().contains("submit"));

The same verbs, three ways

Chained argv, flow files, and the line protocol all build the same Step, so learning one teaches the others and none can drift.

use pixelactions_core::flow::Step;
use pixelactions_core::protocol::{RequestBody, parse_request};
use pixelactions_core::verb::parse_all;

// Chained argv: `pixelactions run --session DIR click:submit type:"hi"`
let steps = parse_all(["click:submit", "type:hi", "scroll:results>-3"])?;
assert_eq!(steps[0], Step::Click { target: "submit".into() });

// The line protocol: `do` names the same action a flow file's `action`
// does, so `{"do":"click"}` is the wire form of `action = "click"`.
let request = parse_request(r#"{"id":1,"do":"click","target":"submit"}"#)
    .expect("a valid request");
assert_eq!(request.id, Some(1));
assert_eq!(
    request.body,
    RequestBody::Step { step: Step::Click { target: "submit".into() } }
);

Parsing a chain is all-or-nothing by design: a typo in step 7 must not perform steps 1 through 6 first.

Speaking the line protocol

protocol holds both directions of the wire format, so you can write a Rust client for pixelactions serve — or an entirely different server that speaks the same thing.

use pixelactions_core::protocol::{PROTOCOL_VERSION, Response, ResponseBody, supported_verbs};

let welcome = Response {
    id: Some(1),
    body: ResponseBody::Welcome {
        version: PROTOCOL_VERSION,
        verbs: supported_verbs(),
        session: "/captures/checkout".into(),
    },
};

// One JSON object per line, newline included — no embedded newlines, in
// either direction.
let line = welcome.to_line();
assert!(line.ends_with('\n'));
assert_eq!(line.matches('\n').count(), 1);
assert!(line.contains("\"result\":\"welcome\""));

The framing rules are the ones LSP, esbuild, and MCP all converged on: one JSON object per line; stdout carries protocol only and stderr is logs; closing stdin is the graceful shutdown; and a version handshake first, so the protocol can change later without breaking programs written against it today.

Reporting what happened

report carries the vocabulary the whole tool reports in — and the distinctions are the point.

use pixelactions_core::report::{RunReport, StepOutcome, StepReport};

// "The OS accepted the event" is not "the app reacted to it".
assert_eq!(StepOutcome::Verified.name(), "verified");
assert_eq!(StepOutcome::Executed.name(), "executed");

// And "it did not work" is not "I declined to try": a refusal is never
// worth retrying, so it earns its own exit code.
let refused = RunReport {
    schema: RunReport::SCHEMA,
    session: "/captures/checkout".into(),
    executed: true,
    steps: vec![StepReport {
        index: 0,
        summary: "click submit".into(),
        outcome: StepOutcome::Refused,
        points: Vec::new(),
        detail: Some("kill switch: the cursor is in a screen corner".into()),
        elapsed_ms: 14,
    }],
};
assert_eq!(refused.exit_code(), 3);

Exit codes are the API: 0 done · 1 a step failed honestly · 2 malformed question · 3 refused.

The modules

Module	What it is
convert	coordinate spaces, per-monitor scaling, screen corners
flow	the flow-file schema and its steps, parsed strictly
plan	resolving labels against a session, totally or not at all
verb	the chained-argv verb:argument grammar
protocol	the serve line protocol, both directions
report	run reports, step outcomes, the exit-code contract
chord	reading cmd+shift+s into modifiers and a key

Relationship to pixelcoords-core

This crate depends on pixelcoords-core for the session schema, and the dependency is one-way and forever — the two tools release independently and neither is pinned to the other’s schedule. Sessions are read through pixelcoords’ own types, which ignore unknown fields, so every additive schema change upstream is a no-op here. Our own config is parsed strictly: tolerance is for other people’s data, not ours.

Stability, honestly. This is pre-1.0 and shares a version with the binary, so a minor bump can change any signature. The flow file, the line protocol, and the exit codes are the parts with a real compatibility promise — the protocol carries PROTOCOL_VERSION and a handshake precisely so it can change without breaking your program. Pin a caret range and read the CHANGELOG before upgrading.

Every example above is compiled and run as a doctest in CI, so nothing on this page can rot silently.

See also

• pixelcoords — the other half: mark regions, get pixel-exact coordinates
• docs/FLOW.md — every step and setting
• docs/PROTOCOL.md — the line protocol, with a working client
• API docs · repository · pixelactions.dev

MIT licensed.

Modules

chord

Reading a key chord: cmd+shift+s → modifiers, then the key.

convert

Coordinate-space conversion — the part most tools get wrong.

display

Which display server this session is actually running.

flow

The flow file: a list of steps referencing a pixelcoords session by label, never by raw coordinate.

plan

Resolution: turn a flow plus a session into a concrete plan, or refuse.

protocol

The line protocol: one JSON object per line, in and out.

report

Run reports — what happened, in a shape a machine can read.

stream

Placing a session’s physical pixel inside a Wayland input region.

verb

verb:argument — the chained-argv form of a step.

virtualdesk

Normalizing a session’s physical pixel into the grid Windows takes.