allwright-surface-mobile-android

Android plugin for the allwright mobile surface.

Current design direction:

• Use a native Rust runtime that drives Android through adb and the built-in UI hierarchy dump/input tools.
• Support native Android views, hybrid apps, and browser-wrapped/WebView-backed apps as first-class targets.
• Mirror the web surface contract: clients send a connect command with Android-specific options and receive a session plus initial page handle back.
• Resolve connect through ADB device discovery: use the requested device name or serial when provided, otherwise attach to the first available Android device or emulator.
• Keep page-scoped commands aligned with web. The first implemented cut is connect, launch_app, click_element, and fill_element.
• Reuse the same selector transport convention clients already know: css=... and xpath=..., including chained locator segments.
• Accept UiAutomator2-style selector strategies directly from clients, including text=..., textContains=..., textStartsWith=..., textMatches=..., description=..., descriptionContains=..., descriptionStartsWith=..., descriptionMatches=..., className=..., classNameMatches=..., resourceId=..., resourceIdMatches=..., packageName=..., packageNameMatches=..., state flags such as clickable=true or selected=true, plus index= and instance=.
• Bootstrap sessions through connected devices or emulators discovered over ADB.
• Ship a standalone runtime artifact so allwright plugin install mobile-android can download the plugin library from GitHub Releases.

Current local smoke path:

• connect discovers devices through adb devices -l, selects the named/serial-matched device when provided, and otherwise uses the first available device entry.
• launch_app installs the APK with adb install -r, resolves the package from --app-id or best-effort APK metadata parsing, then starts the app with am start or monkey.
• click_element and fill_element resolve selectors from the dumped Android UI hierarchy and drive interaction through native adb shell input commands.
• allwright plugin install mobile-android installs the standalone runtime library, but a full engine-routed Android session path still depends on the remaining mobile server integration work.
• Run the smoke test with bash scripts/test-mobile-android.sh /path/to/app.apk 'xpath=//*[@text="Login"]' --app-id your.package.name.

Accessibility snapshots

Android app contexts in Rust, Go, Java, Python, TypeScript, and Vitest expose accessibility_snapshot / AccessibilitySnapshot / accessibilitySnapshot, using the same options as web (format, mode, timeout). JSON is the default; YAML quotes strings to preserve their types in YAML 1.1 and 1.2.

The version-1 documents array contains the app context ID, native package/activity, platform, optional AI snapshot ID, and an application root. Each node has role, name, states, properties, and children. Native properties include class, resource ID, bounds and absolute XPath. Roles map common Android widget classes; names prefer content description over text. Password names are redacted. Unknown/custom classes remain generic. UI Automator exposes only a subset of accessibility semantics; this is not a browser ARIA tree or a full WebView DOM snapshot.

• default: retain the native hierarchy and exposed semantics.
• codegen: the same native tree in standard serialization.
• autoexpect: omit zero-area nodes while retaining their rendered descendants. Disabled nodes with bounds remain present.
• ai: add synthetic aria-ref fields on enabled, positive-area clickable, focusable or editable nodes.

No application source, live attributes, or dumped XML is modified. Use a returned reference with app.locator("ref=<id>") for click, fill, focus, key press, count, text reads, and waits. References must be standalone selectors. Explicit positional XPath, such as xpath=/hierarchy/node[1]/node[2], is also supported; positions are one-based XML sibling positions, independent of Android's index attribute.

The plugin caches references by connection and app context, mapping each to an absolute XPath. Every reference lookup dumps fresh XML and compares the complete hierarchy and foreground package/activity before resolving the path. A mismatch invalidates the generation permanently; capture another AI snapshot to obtain new references. Unchanged AI snapshots reuse references. App close clears the cache; entries expire after ten minutes and the oldest are evicted above 128 app contexts. References are not durable identifiers: identical recycled nodes and screen changes between dump and input cannot be detected atomically.

The engine keeps lazily loaded libraries resident to preserve plugin state. Restart it after rebuilding/replacing a loaded plugin.

Run cargo test -p allwright-surface-mobile-android -p allwright-plugin-sdk --offline for unit coverage. For the simulated-device transport test, first build with cargo build -p allwright -p allwright-surface-mobile-android, then run ALLWRIGHT_TEST_MOBILE_SNAPSHOT=1 bun test typescript/core/tests/mobile-snapshot.integration.test.ts from the repository root. This exercises gRPC and the actual dynamic library, with a deterministic ADB fixture instead of a connected device.