allwright-surface-web

Installable web surface plugin for the allwright engine.

Protocol Rule

This is a hard rule for all future web-plugin development: Chromium web element operations must execute through WebDriver BiDi. This includes click, hover, focus, fill, key input, selector checks, text reads, highlighting, and screenshots.

CDP is permitted only for Chromium browser and tab lifecycle, bootstrapping the Chromium BiDi mapper, and transporting commands to that mapper. It must not be used to inspect the DOM or dispatch user input in a normal web automation path.

JavaScript dialogs

Register page.registerHook(hooks.dialog) before clicking an alert/confirm/prompt trigger, then call hook.wait() and dialog.accept(), dialog.accept("text"), or dialog.dismiss(). Results expose type, message, and defaultValue. Omitted text preserves the default; empty text clears it. Only prompts accept text. One pending hook/dialog per page is supported; hooks are one-shot and handling is never retried. The page-scoped subscription stays alive between hooks so synchronous consecutive dialogs are buffered instead of lost. Page/browser cleanup drains consecutive prompts before releasing that subscription. Waiting, accept, and dismiss support command timeout options; registration does not because cancelling subscription setup can orphan browser state. Dialogs left unhandled remain open and block page work.

src/dialogs.rs subscribes and handles through BiDi on both browsers. Input commands yield when their context opens a prompt, allowing the sequential hook flow to handle it. This includes script-backed press, fill, focus, and Firefox hover actions, not just pointer dispatch. Query/read scripts never use this interruption behavior. Focus keeps the normal semantic/chained selector resolver. This does not cover navigation-triggered/before-unload dialogs. Run the public transport regression with ALLWRIGHT_TEST_BROWSER=chromium or firefox and bunx vitest run typescript/vitest/tests/dialogs.spec.ts after building/installing the local engine and plugin.

Accessibility snapshots

page.accessibilitySnapshot({ format: "json" | "yaml" }) returns a serialized string. JSON is the default. Both formats encode the same ordinary mappings, arrays, strings, booleans, and numbers; role/name/state are separate fields. There are no Playwright snapshot keys, regular expressions, or custom YAML tags to interpret. All YAML string keys and values are quoted so YAML 1.1 and 1.2 parsers preserve names such as yes, on, and date-like strings. Use JSON.parse, json.loads, or a standard YAML parser.

const snapshot = JSON.parse(await page.accessibilitySnapshot());
const yaml = await page.accessibilitySnapshot({ format: "yaml", timeoutMs: 10_000 });

Other clients expose the same operation:

The JSON Schema describes version 1. For example:

version: 1
documents:
  - contextId: page-context
    parentContextId: null
    url: https://example.test/
    title: Example
    root:
      role: document
      name: Example
      states: {}
      properties: {}
      children:
        - role: checkbox
          name: Accept terms
          states:
            checked: mixed
          properties: {}
          children: []

documents contains the top page followed by its iframe documents in depth-first order. parentContextId links each frame document to its parent; context IDs are opaque browser session identifiers, not selectors or persistent element IDs. Each document is collected in its own browsing context, including cross-origin frames. Frames retain iframe nodes in their parent's DOM tree, while their contents appear once in documents. Frame documents are collected independently, including frames whose embedding element is hidden. A detached frame or script failure fails the command instead of silently returning an incomplete snapshot; normal command timeout/retry options apply. Frames are sampled sequentially, so this is not an atomic snapshot of a page that is changing during collection.

Nodes have role, accessible name, states, properties, and ordered children. Ordinary text uses role: text. Containers without accessibility semantics are flattened. States include native and explicit ARIA checked, selected, expanded, pressed, disabled, required, readonly, heading level, and numeric range values where available. Undefined states are omitted, explicit false remains a boolean, and mixed check states are strings. Properties include accessible descriptions, control values, placeholders, link URLs, and ARIA metadata. Password and file input values are omitted.

The collector walks the composed DOM, including open shadow roots, assigned slots, and aria-owns relationships with cycle/duplicate protection. By default it excludes hidden and inert subtrees within each document, but includes off-screen content. Referenced hidden labels can still contribute to accessible names. Collection and serialization live in the web plugin, and execution uses WebDriver BiDi for both Chromium and Firefox.

This is a DOM-derived accessibility representation, not the browser's native platform accessibility tree or an accessibility conformance audit. Closed shadow roots, browser-internal widgets, canvas internals, and accessibility semantics exposed only through ElementInternals are not available to this DOM walker. Native role/name behavior can differ from the browser's accessibility engine. Role, name, description, and visibility computation use Allwright-owned accessibility semantics. Playwright and W3C specifications serve as behavioral references; no external accessibility library or copied Playwright code is bundled. This independent implementation does not claim Playwright-equivalent maturity or complete specification conformance.

Validation:

cargo test -p allwright-surface-web --lib
cargo test -p allwright-surface-web --test accessibility -- --ignored
ALLWRIGHT_TEST_BROWSER=firefox cargo test -p allwright-surface-web --test accessibility -- --ignored

The opt-in browser test launches a temporary browser profile and a local fixture server. It covers structured JSON/YAML equivalence, labels/descriptions, typed states, hidden content, password values, shadow DOM/slots, ARIA ownership cycles, off-screen content, and same-origin/cross-origin frames.

Snapshot modes are independent of the output format:

const snapshot = JSON.parse(await page.accessibilitySnapshot({ mode: "ai", format: "json" }));
// For a node in the current document:
await page.click(`[aria-ref="${node["aria-ref"]}"]`);

References are opaque strings, unique across documents with a random document prefix. They stay stable while the element, role, and accessible name remain unchanged. Query them in the snapshot document's frame (or within its open shadow root); a plain document CSS query does not cross frame or shadow boundaries. Text nodes and non-rendered/non-pointer elements have no reference. A reference does not guarantee an element is enabled, unobscured, or currently actionable. On the next AI capture, stale attributes assigned by the collector are removed; other modes leave existing attributes alone. Navigation creates a new reference registry. The collector owns aria-ref on referenced elements and replaces any existing value there. DOM changes can invalidate references; capture again when needed. These modes take behavioral inspiration from Playwright but do not claim identical tree output.

Iframes as pages

Resolve an iframe locator to a page using await page.locator("iframe").Frame({ timeoutMs: 10_000 }) (TypeScript; .frame() is also supported). Go uses locator.Frame(ctx, CommandOptions{Timeout: ...}), Rust uses locator.frame_with_options(...), and Java/Python use locator.frame(options). The returned page supports ordinary locators and can resolve nested frames the same way.

Resolution requires exactly one iframe, waits for its document to reach readyState = complete and remain free of DOM mutations for 200 ms, and retries within the command timeout (default 10 seconds). This is a document readiness check, not a network-idle guarantee. Cross-origin frames are resolved through WebDriver BiDi window values; their contents are accessed in the child browsing context, without reading the child DOM from the parent. Chromium uses the existing BiDi mapper transport; Firefox uses native BiDi. Closing a frame page releases its Allwright session without closing the containing tab or removing the iframe. A detached frame handle fails; resolve the locator again after replacement.