mobile-agent-protocol

The wire protocol between the host daemon and every device agent (design D4, spec device-protocol): the message schema, framing, the handshake and capability negotiation, per-stream event sequences and liveness. It holds no device code. The Android agent's Kotlin codec in agent-android/src/dev/mobileagent/agent/codec writes the same bytes; golden frames checked in both directions keep the two in step.

Layout

src/
  message.rs     envelope: hello, welcome, request, response, ping, pong; methods, parameters, results
  snapshot.rs    snapshot payload: string table, windows and their metadata, nodes, bounds, flags, scroll semantics
  event.rs       events and their streams
  error.rs       structured errors and the code table
  capability.rs  the capability set
  frame.rs       length-prefixed framing, oversized-frame handling
  session/       HostSession (daemon end) and AgentSession (agent end), without I/O
  sequence.rs    per-stream sequence tracking
  liveness.rs    ping and silence timing
  cbor.rs        shared decoding helpers
golden/          frames this crate writes for the golden samples
golden/kotlin/   frames the Kotlin codec writes for the same samples
tests/           one file per task: schema, framing, handshake, events, liveness, golden, bench_inputs
benches/         codec.rs: microbenchmarks of the codec and the framing; support/: their seeded inputs and the counting allocator

cargo bench -p mobile-agent-protocol -- --noplot [filter] runs the benchmarks of design D14 of the microbenchmarks change with criterion's default settings, about 10 s each, and prints each one's allocations per iteration as allocations <id> <allocations> <bytes>. A number that goes into a report comes from a run on a quiet host (design D15): just micro waits for one and compares every suite with reports/micro/baseline.json.

Frames

A frame is a 32-bit big-endian payload length followed by exactly one CBOR item (RFC 8949) of that length, with no bytes after the item. The limit applies to the payload length. Before the handshake each side uses its own maximum; afterwards both use the negotiated one, the smaller of the two.

• The sender refuses a message over the limit and writes none of it (write_frame). An agent answers such a response with a frame_too_large error naming the limit and the request instead.
• The receiver reports a frame whose header exceeds the limit (Oversized), with the kind, request id and method read from its first 32 bytes, and drops its payload as it arrives: nothing of it is delivered and the next frame decodes normally. The host turns it into a frame_too_large error for the pending request; the agent answers an oversized request the same way. Bytes that only feed such a discard do not count as signs of life, so a corrupt length cannot keep a dead peer alive.

CBOR layout rules

Both codecs follow these rules, so the same message is the same bytes on both sides.

1. Preferred serialization. Every integer and length head takes its shortest form.
2. Definite lengths only. Arrays, maps, text and byte strings always carry their length. Neither encoder writes indefinite-length items, and every golden frame is checked for them. The Kotlin decoder rejects them; the Rust decoder, like minicbor's derived decoders, accepts them.
3. Maps keyed by field index. Messages (except events), parameters, results, errors, snapshots, windows, nodes and scroll semantics are maps whose keys are unsigned integers, written in ascending order, each once.
4. Absent optional fields are omitted, never written as null. Decoders read a null as absent. A node's child list is omitted when empty and a window's modal flag when false; every other list is always present, possibly empty.
5. Positional arrays have no optional elements. Events, bounds, scroll ranges and scroll positions are arrays with a fixed layout, so their length never shrinks. Receivers skip elements past those they know, so new elements may be appended.
6. Unknown keys are skipped, and a repeated key's last value wins.
7. Types. Unsigned fields are major type 0, signed ones (coordinates, bounds, window layer, scroll range and position) major type 0 or 1, booleans are simple values, text is UTF-8 (decoders reject invalid UTF-8), image data is a byte string, flags are u32 bitmasks and enums small integers. No floats and no tags.
8. Parameters and results are always maps. A request carries its parameters even when empty ({}); a successful response carries its result, {} for methods with none.
9. Nesting. Nodes nest at most MAX_NODE_DEPTH (256) levels below a window root; decoders reject deeper trees. The Rust decoder keeps its stack of open nodes on the heap: the derived recursive decoder took about 20 KiB of stack per level in a debug build and overflowed a 2 MiB thread below 128 levels.

Receivers never allocate more than the input can hold: array lengths are capped by the bytes left before anything is reserved.

Messages

The kind is key 0 of a map or element 0 of an event array. ? marks an optional key.

An error is {0: code text, 1: message text, 2: retryable bool}. The response echoes the method so its result decodes without the request at hand. A decoder reports an unknown kind as such, and the sessions skip it.

Methods

An agent answers a method it does not know with unknown_method; the request still decodes, its parameters skipped, so it can be answered.

A snapshot's evict_event_less (task 3.7), when true, asks the agent to evict its platform cache of the windows that send no change events (emits_events false, below) before reading them. Input the agent did not inject, such as the emulator controller's, leaves only those windows' cache stale, since every other window's events keep its cache current; the eviction refreshes them without the resync of the whole tree stream, which also needs the events capability. An agent that does not know the key skips it and reads with its cache.

A snapshot's evict (task 10.11) names windows, by id, whose platform cache the agent evicts before reading them. Views an app moves without a change event keep the bounds the cache read before the move; the host names the windows whose nodes a bounds probe found moved.

bounds (task 10.11) reads the bounds of nodes of the agent's last snapshot again, by the ids that snapshot gave them, past the platform's node cache, one round trip to the app per node (AccessibilityNodeInfo.refresh() on Android). The result lists each node still on screen as [node, bounds], in the request's order; a node that is gone, or that the last snapshot did not hold, is left out. A host without frames to watch probes a few anchor nodes while it waits for the screen to settle (design D8). The method needs the bounds capability, so an agent from before it is never asked.

A swipe's rest_ms (task 10.16) asks the agent to stop the pointer before it lifts: the pointer rests at (x2, y2) that long, moves one pixel on along the line and lifts there. Android's velocity tracker takes a pointer that has not moved for 40 ms as stopped when its next event comes, so the content stops where the pointer left it instead of flinging on at the pointer's speed; scroll --until strokes this way, so that each stroke carries the content only as far as the pointer went. An agent that does not know the key skips it and lifts the pointer moving; the daemon deploys the agent built beside it, which knows it.

Snapshot

class, res_pkg, package and owner index the per-snapshot string table. truncated says which request bound cut the walk short (bit 0 depth, 1 nodes, 2 windows). seq is the last tree-stream event the agent had sent when it read the tree. Window kinds are AccessibilityWindowInfo.TYPE_* values. Node flag bits: clickable, long_clickable, focusable, focused, enabled, checkable, checked, scrollable, selected, editable, password, visible, accessibility_focused, modal, on_top (bits 0 to 14); modal marks a view the platform keeps assistive technology inside, as iOS's accessibilityViewIsModal does, leaving out its siblings and what they hold, and on_top a node drawn after every sibling its bounds overlap, as Android reports the order a parent draws its children in (getDrawingOrder), which the order of the children does not say. Window flag bits: active, focused, accessibility_focused, picture_in_picture, transformed (bits 0 to 4); transformed marks a window drawn away from where its layout puts it, partly off the display or at another size than its layout's (the latter from Android 14), as the platform draws a window it animates in or out.

Keys 8 to 10 of a window are what the agent knows about the window beyond the platform's own window states (task 3.6):

• emits_events says whether the window's toolkit sends change events for its content. false marks a window whose changes are seen only by reading it again: Jetpack Compose sends none to a UI automation client (spike S1), so a consumer re-reads such a window after each action and on a poll. Absent when the agent cannot tell.
• owner names the system process that owns the window, such as com.android.systemui for the status bar or the permission controller for a permission prompt. Absent when the application under automation owns it.
• modal marks a window, such as a dialog, that keeps the user from the windows below it until it is dismissed. Omitted when false.

Key 9 of a node is present on nodes that scroll. axis is 0 unknown, 1 vertical, 2 horizontal. Flag bits: back (0), toward the start of the content (up, or left in a left-to-right layout), and forward (1), toward its end, each set when the node can scroll that way from where it is. range is the span of the content in unit, both ends inclusive: 0 items, a collection's items along the axis numbered from 0 ([0, 0, 199] for 200 rows), or 1 percent of the scrollable range, 0 to 100, for a position the platform reports as a value between bounds. position, present only with a range, is the part of it inside the viewport, both ends inclusive: [0, 15] while the first 16 rows are in view, [37, 37] at 37 percent. A platform that reports neither leaves them out and the flags say what it knows: Compose reports only the rows inside a list's viewport, so a consumer tells what lies beyond them from these semantics rather than from nodes. Values of axis, unit and flag bits this version does not define survive a round trip.

Events and streams

Every stream numbers its events from 1 on each connection. On a lossless stream each event is one more than the last; any other number marks the stream out of sync, the host drops its events and asks the agent to resync it, and the stream is in sync again from the seq the answer names. When the agent fails that automatic resync with a retryable error, such as busy, the host sends it again after session::RESYNC_BACKOFF, a wait that doubles with every failure in a row up to session::RESYNC_BACKOFF_MAX, until the stream is in sync, its events dropped meanwhile (task 3.7). A failure no retry fixes, and any failure of a resync asked for with HostSession::resync, is reported as HostEvent::ResyncFailed, and the stream stays out of sync. A lossy stream only increases; the numbers it skips are counted as dropped frames. An agent that drops an event from a full queue skips its number (AgentSession::skip_event) so the host sees the gap. Unknown streams are tracked as lossless.

Capabilities

A u32 bitmask: tree (bit 0), input (1), screenshot (2), frames (3), events (4), text_set (5), bounds (6). Unknown bits survive a round trip. The host refuses a request whose method needs a capability the agent did not advertise, naming it, and sends nothing; the agent refuses such a request too.

Errors

The flag travels with every error; Error::new takes it from this table.

Sessions

HostSession and AgentSession are state machines without I/O. The caller feeds bytes to receive, writes what poll_transmit returns, calls handle_timeout at poll_timeout, and reads poll_event. Every time-dependent method takes now; the sessions never read the clock, so tests drive time themselves.

1. The agent speaks first: its hello, ready as soon as the session exists, doubles as proof that something listens behind an adb forward.
2. The host selects the highest version both speak and the smaller maximum frame size, checks the per-launch token if it has one, and answers with a welcome. On failure the welcome carries the error (no common version lists both version sets and asks to redeploy the agent) and the host closes.
3. Requests carry host-assigned ids, unique among those in flight; responses may arrive in any order.
4. A request may carry a relative deadline. The agent answers it with deadline_exceeded when the deadline passes and drops the late answer; the host reports the error itself when nothing arrives within the grace period after the deadline. The connection stays usable.
5. After a second without hearing from the other side a session pings; after three it closes as Silent, failing pending and later requests with disconnected and a reconnect hint. The end of the stream closes it at once.

Defaults

Golden frames and the Kotlin codec

tests/common/samples.rs defines 41 samples: one per message kind, parameter and result type, with integers at every head-length boundary, u32 and u64 values past 2^{31 and 2}63, multi-byte UTF-8, a byte string and a 25-element array, plus window metadata in every state (response_snapshot_window_meta), scroll semantics in every shape (response_snapshot_scroll), a snapshot request evicting event-less windows (request_snapshot_evict), one evicting windows by id (request_snapshot_evict_windows) and a bounds probe with its result (request_bounds, response_bounds). golden/<name>.bin is the frame this crate writes for each. agent-android/codec-test/Samples.kt builds the same values in Kotlin.

cargo test -p mobile-agent-protocol                                   # includes the golden checks
PROTOCOL_UPDATE_GOLDEN=1 cargo test -p mobile-agent-protocol --test golden   # rewrite golden/ after a deliberate change
agent-android/codec-test.sh --write-golden               # Kotlin checks on the host JVM and a device; rewrites golden/kotlin/
agent-android/codec-test.sh --host-only                  # the host JVM only

codec-test.sh compiles the codec and its test with the flags of agent-android/build.sh, adds the dexed classes to the same jar, runs it on the host JVM and then once through app_process on the connected device. It checks that every Kotlin sample encodes to the Rust frame byte for byte, that every Rust frame decodes and re-encodes to itself, and that the Kotlin decoder enforces the rules above. The Rust tests decode golden/kotlin/ to the same messages.

Interpretations of the spec

• Capability advertisement says the agent advertises its capabilities after version negotiation. They travel in the same hello as its versions, as the research note and D4 put it, and the host acts on them only once it has selected a version.
• Event channel asks for events on a stream separate from responses. Event streams are logical: event frames, told apart from responses by their kind, on the same connection.
• deadline_exceeded is retryable: the spec defines the flag as whether retrying the same request is meaningful, and the agent stays responsive, so the same request can finish another time.

Spec requirements and their tests