framewatch

Event-driven, change-triggered window capture that emits timestamped screenshots + metadata so an AI coding agent can reconstruct what happened — without a continuous frame stream.

Why

Continuous screen capture floods an agent's context with near-duplicate frames. On-demand "screenshot now" misses everything in between. framewatch sits in the middle: it watches one window and only writes a frame to disk at semantically meaningful moments:

• the window settles after a burst of change (the final, important state),
• a spinner / busy indicator starts or stops animating,
• a volatile region (counter, progress %, log tail) is sampled on a throttle,
• plus the initial frame and any manual trigger.

Everything else is collapsed. Between saved frames an append-only timeline records what happened (how many frames were coalesced, how long the window was busy, which regions changed), so an agent reads a compact, timestamped story and opens only the images that matter.

Detection is heuristic and runs in the hot path with no LLM call. You can pre-annotate where spinners and volatile values live via the GUI, which makes detection both faster and more accurate — or set auto_detect_spinners = true to have a small, compact loading animation detected and collapsed automatically, no ROI hint required.

What makes it new

The primitives exist; the assembled product did not. No crate combined change-triggered capture + transient-state (spinner/volatile) awareness + timestamped, agent-readable artifacts in one importable package:

Using framewatch from another project or agent? See docs/AGENT_INTEGRATION.md for the exact where/how-to-call contract, and dist/framewatch.json for a machine-readable manifest.

Codex integration

This repository includes both Codex-native discovery layers:

• AGENTS.md gives Codex the project architecture, public-contract rules, and verification commands when it works on Framewatch itself.
• .agents/skills/framewatch/SKILL.md teaches Codex when to use shot, watch, or record, and how to consume the resulting artifacts. Codex discovers the skill automatically in this checkout; invoke it explicitly with $framewatch or let it trigger for desktop UI capture tasks.

To use the skill from another repository, ensure a current framewatch.exe is on PATH and copy the framewatch skill directory into that repository's .agents/skills/ directory (or into your user-level ~/.agents/skills/). See the Codex skill documentation for skill discovery details.

Install

# CLI (Windows live capture):
cargo install framewatch --features wgc        # add gui for the picker: --features "wgc gui"

# CLI with narrated recording (also enables wgc; requires ffmpeg on PATH):
cargo install framewatch --features "wgc record"
framewatch transcriber setup   # optional preflight; record also runs this automatically

# As a library (engine + sinks only, no clap/egui):
cargo add framewatch --no-default-features

Platforms. The detection engine is platform-agnostic and compiles/tests everywhere. The live capture backend is Windows-only (Graphics Capture API) and is enabled with the wgc feature. macOS/Linux backends are a documented future extension — they just implement the same CaptureBackend trait.

Quickstart (CLI)

# 1. See what can be captured
framewatch windows

# 2. Pick a window + mark spinner/ignore regions visually
framewatch gui

# 3. Or run headless by title
framewatch watch --title "Visual Studio Code" --out ./.framewatch

# 4. Or against a saved config
framewatch watch --config framewatch.toml

Record & narrate → an LLM package (V4)

Sometimes you don't want a deduped story — you want to show and tell. The record subcommand (install with --features "wgc record", needs ffmpeg on PATH) continuously records one window to video while you narrate into the microphone, then transcribes the narration locally and bundles everything an LLM needs to act on it. On first use, Framewatch automatically downloads a pinned, checksum-verified whisper.cpp runtime and base.en model (~150 MiB) into the user cache; later recordings reuse it:

# Record a window for 60s (or stop early with Ctrl+C) while you talk.
framewatch record --title "My Game" --duration 60

It writes a package directory:

recording.mp4         # the window video (your narration muxed in)
audio.wav             # the raw mic narration
transcript.json/.srt  # segments with start_ms/end_ms from video start
recording.json        # manifest
PROMPT.md             # the prompt to hand the model (transcript inline)
README_FOR_AGENT.md

Because every transcript segment is timestamped from the start of the video, a model can correlate "click this" with the exact on-screen moment — ingesting recording.mp4 directly or pulling a frame with ffmpeg -ss <seconds> -i recording.mp4 -frames:v 1 frame.png. See the recording-package contract.

Transcription uses managed whisper.cpp by default. Use --no-transcribe for an audio-only package with no model download. Advanced users can override the engine with --transcribe-cmd; {audio} and {output} are substituted and the command writes framewatch transcript JSON or SRT. Set FRAMEWATCH_WHISPER_DIR to change the managed cache parent. The runtime and model are downloaded rather than embedded in the crate, keeping installation small while making the default workflow self-configuring.

No microphone? Recording degrades gracefully — it warns and produces a video-only package (no transcript). Pass --no-audio to opt out of mic capture explicitly.

The agent-consumption contract

A session directory (./.framewatch/<session_id>/) contains:

frames/000000_initial.png
frames/000003_settled.png
timeline.jsonl          # one JSON event per line, chronological
session.json            # manifest: target, time range, config, ROI hints, counts
README_FOR_AGENT.md     # how to read this directory

An agent should: read session.json, stream timeline.jsonl, and open only the PNGs referenced by kind:"settled" / kind:"busy_end" unless it needs finer detail. coalesced_frames tells it how much activity each saved image represents.

A whole "run tests" workflow collapses to ~4 timeline entries and 2 images instead of ~75 screenshots:

{"seq":0,"kind":"initial","elapsed_ms":0,"image":"frames/000000_initial.png","note":"Session start."}
{"seq":1,"kind":"busy_start","elapsed_ms":1200,"image":null,"note":"Test runner started (spinner active)."}
{"seq":2,"kind":"busy_end","elapsed_ms":4830,"image":"frames/000002_busy_end.png","coalesced_frames":71,"note":"Spinner stopped after 3.63s; 71 animation frames collapsed."}
{"seq":3,"kind":"settled","elapsed_ms":5180,"image":"frames/000003_settled.png","note":"Settled: test results rendered."}

Embedding

use framewatch::{Config, Target, DirectorySink, Engine, CaptureBackend, ControlFlow, Sink, SystemClock};

fn main() -> anyhow::Result<()> {
    let config = Config::builder()
        .target(Target::ByTitleRegex("Visual Studio Code".into()))
        .out_dir("./.framewatch")
        .settle_ms(350)
        .spinner_roi("test-runner", [0.02, 0.94, 0.04, 0.05])
        .ignore_roi("clock", [0.92, 0.0, 0.08, 0.03])
        .build()?;

    let mut engine = Engine::new(config.clone(), SystemClock);
    let mut sink = DirectorySink::new(&config)?;

    // On Windows (built with `--features wgc`), use the live backend:
    // let mut backend = framewatch::default_backend(&config)?;
    // Off-Windows / in tests, drive the engine with your own frames:
    let mut backend = framewatch::MockBackend::new(vec![/* RawFrames */]);

    backend.run(&mut |frame| {
        for event in engine.process(&frame, frame.captured_at) {
            sink.on_event(&event).ok();
        }
        ControlFlow::Continue
    })?;
    Ok(())
}

The Engine is pure: (state, RawFrame, now) -> Vec<CaptureEvent>. It does no I/O, no capture, and takes its clock by injection — which is why the whole detection pipeline is unit-tested without a GPU, screen, or Windows.

Cargo features

The core library pulls no Windows or GUI deps unless you opt in.

License

Licensed under either of MIT or Apache-2.0 at your option.