pixelcoords-core

The platform-free core of pixelcoords: the geometry, the session.json schema, coordinate spaces and units, template relocation, point verdicts, click-point resolution, region diffing, and click-code generation — with no window system, no capture backend, and #![forbid(unsafe_code)].

Want the tool? Install the binary: cargo install pixelcoords. Want to build something on top of it? That's this crate.

[dependencies]
pixelcoords-core = "0.5"

What you'd use it for

A pixelcoords session is a human's answer to "where is this thing on screen, exactly" — labeled regions, pixel-exact, with crops. This crate is everything you need to consume that answer in Rust: read the schema, resolve a label to the point you should click, ask whether a region is still where it was, or check that a coordinate landed inside it.

That covers most tools built here: test harnesses that assert on screen positions, automation runners, screenshot annotators, converters between DPI spaces, and alternative front-ends over the same session format.

You may not need this crate at all. If you only want to read session.json from another language, the schema is plain JSON and fully documented in docs/OUTPUT.md. Reach for this crate when you want the behavior too — the matching, the scoring, the geometry — not just the data.

The coordinate model, which you must get right

Everything else here is downstream of this, and it is the thing tools get wrong.

• A session stores physical pixels, twice per selection: px is monitor-local, global_px is the same region on the global desktop grid.
• Each MonitorRecord carries origin_px (its position on that global grid) and scale (its DPI factor), so global = monitor.origin_px + local.
• There is no universal logical space, and the schema does not pretend there is. For logical points, divide by the containing monitor's scale — never a global one. Mixed-DPI desktops are the normal case.
• Input APIs disagree about which space they want: macOS CGEvent takes logical points; Windows SendInput and X11 XTEST take physical pixels. Converting to the wrong one clicks the wrong place without erroring.

Read a session and resolve its click points

The core loop of nearly every consumer: for each labeled region, find the point you would actually click.

use pixelcoords_core::session::SessionFile;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
let session: SessionFile = serde_json::from_str(EXAMPLE_SESSION)?;

for selection in &session.selections {
    let monitor = session
        .monitors
        .iter()
        .find(|m| m.index == selection.monitor)
        .expect("a session describes every monitor it references");

    // `click_point` is an interior point of the shape — the centroid for
    // a rect, and a point guaranteed *inside* a triangle or freehand
    // polygon, where the centroid can fall outside the shape entirely.
    let local = selection.px.click_point();
    let global_x = monitor.origin_px.x + local.x;
    let global_y = monitor.origin_px.y + local.y;

    // Logical points, for an API like macOS CGEvent.
    let logical_x = f64::from(global_x) / monitor.scale;
    let logical_y = f64::from(global_y) / monitor.scale;

    println!(
        "{}: physical ({global_x}, {global_y}) · logical ({logical_x}, {logical_y})",
        selection.label
    );
}
# assert_eq!(session.selections.len(), 1);
# Ok(())
# }
#
# const EXAMPLE_SESSION: &str = r#"{
#   "schema": 1,
#   "app": { "name": "pixelcoords", "version": "0.5.3" },
#   "created_utc": "2026-07-29T00:00:00Z",
#   "monitors": [
#     { "index": 0, "name": "Built-in", "primary": true,
#       "origin_px": { "x": 0, "y": 0 },
#       "size_px": { "w": 3600, "h": 2338 }, "scale": 2.0 }
#   ],
#   "selections": [
#     { "shape": "rect", "label": "submit", "monitor": 0,
#       "px":        { "x": 800, "y": 400, "w": 100, "h": 80 },
#       "global_px": { "x": 800, "y": 400, "w": 100, "h": 80 },
#       "crop": "submit.png" }
#   ]
# }"#;

Geometry

Shape covers every tool pixelcoords draws — Rect, Circle, Ellipse, Triangle, and Poly (regular N-gons and freehand alike) — with an untagged serde representation, so shapes round-trip through JSON on their own.

use pixelcoords_core::geometry::{Point, Rect, Shape};

let button = Shape::Rect(Rect::new(800, 400, 100, 80));

assert_eq!(button.click_point(), Point::new(850, 440));
assert!(button.hit_test(Point::new(810, 410)));
assert!(!button.hit_test(Point::new(10, 10)));
assert_eq!(button.bbox(), Rect::new(800, 400, 100, 80));

// Rotation is metadata on rects and ellipses, so hit-testing takes the
// angle rather than mutating the shape.
assert!(button.hit_test_rotated(45, button.click_point()));

Also here: regular_polygon (N-gon construction from a center and a point to aim at), simplify_path (Ramer–Douglas–Peucker, for freehand), rotate_point_about, normalize_deg, and the clamped move/resize helpers the overlay drives.

Line is the measure tool's primitive and deliberately not a Shape — it marks a distance, not a region, so it has no interior to hit-test or crop. It carries length, delta, angle_deg (clockwise from +X, because screen Y grows downward), endpoint and segment hit-testing for grabbing, and constrained for the 45° snap.

use pixelcoords_core::geometry::{Line, Point};

let gap = Line::new(Point::new(0, 0), Point::new(30, 40));

assert_eq!(gap.length(), 50.0);
assert_eq!(gap.delta(), (30, 40));
assert!((gap.angle_deg() - 53.13).abs() < 0.01);

Is this region still where it was?

locate is masked template matching — the engine behind pixelcoords find. Give it a grayscale screen and a template cut from the saved crop, and it reports where that region is now.

Two constants define the trust model, and they matter more than the algorithm: SCORE_FLOOR (0.9) is the minimum normalized correlation to count as found, and AMBIGUITY_GAP (0.03) is how far the best match must beat the runner-up. A template that matches in two places is reported as ambiguous rather than picked between — the caller is expected to refuse, because acting on the wrong instance is worse than not acting.

Template carries an optional mask, so non-rectangular regions match on the pixels the human actually marked instead of the bounding box. report() assembles a Report<FindResult> — the same JSON pixelcoords find prints, with per-label Delta values, so you can tell what moved and by how much.

report::Report<T> is the envelope every scoring command shares: schema, command, captured_utc, an aggregate ok, and the rows. Row-level answers stay on the rows — FindResult::found, Verdict::hit — because a caller usually needs to know which one failed, not merely that one did.

Did this point land in the right place?

verdict::assess answers that in whichever space your point already is — Origin::Global, Origin::Monitor(i), or Origin::Window for --target sessions. An origin says where (0, 0) is; space::Units answers the separate question of whether a coordinate is in device pixels or logical points.

use pixelcoords_core::geometry::Point;
use pixelcoords_core::session::SessionFile;
use pixelcoords_core::space::Origin;
use pixelcoords_core::verdict::assess;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
# let session: SessionFile = serde_json::from_str(EXAMPLE_SESSION)?;
let verdict = assess(&session, Point::new(850, 440), Origin::Global, None)?;

assert!(verdict.hit);
assert_eq!(verdict.contained_in[0].label, "submit");

// A miss still reports what you nearly hit, and how far off you were.
let miss = assess(&session, Point::new(1600, 440), Origin::Global, None)?;
assert!(!miss.hit);
assert_eq!(miss.nearest.expect("a nearest region").region.label, "submit");
# Ok(())
# }
#
# const EXAMPLE_SESSION: &str = r#"{
#   "schema": 1,
#   "app": { "name": "pixelcoords", "version": "0.5.3" },
#   "created_utc": "2026-07-29T00:00:00Z",
#   "monitors": [
#     { "index": 0, "name": "Built-in", "primary": true,
#       "origin_px": { "x": 0, "y": 0 },
#       "size_px": { "w": 3600, "h": 2338 }, "scale": 2.0 }
#   ],
#   "selections": [
#     { "shape": "rect", "label": "submit", "monitor": 0,
#       "px":        { "x": 800, "y": 400, "w": 100, "h": 80 },
#       "global_px": { "x": 800, "y": 400, "w": 100, "h": 80 },
#       "crop": "submit.png" }
#   ]
# }"#;

contained_in lists every region containing the point, in stacking order, so overlaps are disambiguated by the caller rather than silently by the library.

Generate click code

use pixelcoords_core::emit::{EmitFormat, Platform, emit};
use pixelcoords_core::session::SessionFile;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
# let session: SessionFile = serde_json::from_str(EXAMPLE_SESSION)?;
let snippet = emit(&session, EmitFormat::Pyautogui, Platform::MacOs, None)?;
assert!(snippet.contains("pyautogui"));
# Ok(())
# }
#
# const EXAMPLE_SESSION: &str = r#"{
#   "schema": 1,
#   "app": { "name": "pixelcoords", "version": "0.5.3" },
#   "created_utc": "2026-07-29T00:00:00Z",
#   "monitors": [
#     { "index": 0, "name": "Built-in", "primary": true,
#       "origin_px": { "x": 0, "y": 0 },
#       "size_px": { "w": 3600, "h": 2338 }, "scale": 2.0 }
#   ],
#   "selections": [
#     { "shape": "rect", "label": "submit", "monitor": 0,
#       "px":        { "x": 800, "y": 400, "w": 100, "h": 80 },
#       "global_px": { "x": 800, "y": 400, "w": 100, "h": 80 },
#       "crop": "submit.png" }
#   ]
# }"#;

Platform exists because the same coordinate means different things per tool: pyautogui makes its process DPI-aware on import, so it addresses physical pixels on Windows but logical points on macOS. cliclick and xdotool each exist on one platform and don't branch.

The modules, and how much they move

Stability, honestly. This stays on 0.x and shares a version with the binary, so a minor bump can change any signature here. Two things carry a real compatibility promise, and they are versioned separately because they version different things: the session schema on disk (session::SCHEMA_VERSION, still 1) and the documents the commands print (report::CLI_SCHEMA_VERSION, now 2). Additions to either are optional fields, and consumers that ignore unknown keys keep working. The top rows are what tools are actually built on; the overlay internals exist to serve the binary and move with it. Pin a caret range and read the CHANGELOG before upgrading.

Every example above is compiled and run as a doctest in CI, so nothing on this page can rot silently.

See also

• pixelcoords — the tool this serves
• pixelactions — the executor half: consumes these sessions and performs the interactions
• API docs · repository · pixelcoords.dev
• docs/OUTPUT.md — every JSON shape this crate produces

MIT licensed. Bundled font: JetBrains Mono, OFL 1.1.