pub enum Selector {
Text {
text: Pattern,
modifiers: Modifiers,
},
Id {
id: String,
modifiers: Modifiers,
},
Label {
label: String,
modifiers: Modifiers,
},
Role {
role: Role,
name: Option<Pattern>,
modifiers: Modifiers,
},
Focused {
focused: True,
},
Anchor {
anchor: AnchorBox,
index: IndexModifiers,
},
LocalizedText {
localized_text: BTreeMap<String, String>,
modifiers: Modifiers,
},
OcrText {
ocr_text: String,
locales: Vec<String>,
modifiers: Modifiers,
},
AnchorRelative {
anchor: Box<Selector>,
dx: f64,
dy: f64,
},
Point {
nx: f64,
ny: f64,
},
Fallback {
fallback: Vec<Selector>,
},
}Expand description
Selector — 6 mutually exclusive base forms.
Each variant carries its base-shape fields and the modifier set
stackable on that base. serde(untagged) ensures the JSON wire form
matches the existing TS Selector wire 1:1 (no {type: "text", ...}
discriminator — the presence of text / id / label / role /
focused / anchor is the discriminator, mirroring TS’s
isXSelector type-guards).
Variants§
Text
{ text: 'hello' | /^hi/, ...modifiers }
Fields
Id
{ id: 'btn-xyz', ...modifiers }
Fields
Label
{ label: 'Settings', ...modifiers }
Fields
Role
{ role: 'button', name?: pattern, ...modifiers }
Fields
Focused
{ focused: true } — no modifiers (focus is runtime-resolved).
Anchor
{ anchor: { ...spatial keys }, ...indexModifiers }
Fields
index: IndexModifiersIndex pick applied after spatial filtering.
LocalizedText
{ localized_text: { en: "Submit", ja: "送信", es: "Enviar" }, ...modifiers }
— v5.18 c1 (a11y-i18n initiative L4 layer). Per-locale text table for
elements whose visible text varies across locales when no stable
accessibilityIdentifier is exposed. Adapter desugars this variant to
Selector::Text { text: <picked-by-current-locale> } before passing
to the resolver — the resolver itself never sees LocalizedText
(unreachable arm in resolver match). Locale is read from the last
launchApp -AppleLanguages argument the adapter parsed; fallback to
“en” when current locale not in table (or no launchApp seen yet).
Fields
OcrText
{ ocr_text: "Submit", locales: ["en"], ...modifiers } — v5.19 c1
(a11y-i18n initiative L5 layer). Apple Vision OCR keyword over the
current XCUIScreen screenshot. For elements with no stable
accessibilityIdentifier AND no a11y label (custom-drawn UI / 3rd
party libs that bypass UIAccessibility). Adapter handles dispatch
directly (find_by_text_ocr → IOHID synthesize at frame center),
bypassing the standard resolver pipeline — OcrText never reaches
resolver match arms. Locales are BCP-47 subtags; empty array
defaults to adapter’s last_locale (which itself defaults to “en”).
Fields
ocr_text: StringSubstring keyword to OCR-match against text observations.
Case-insensitive substring match on topCandidates(1) of each
VNRecognizedTextObservation.
AnchorRelative
{ anchored: { anchor: <selector>, dx: -0.15, dy: 0 } } — v5.20 c1
(a11y-i18n initiative L6 layer). Anchor-relative coord for icon-only
/ pure-graphic elements (~15-20% of “lib without a11y” scenarios)
where no sense layer can locate the target directly, but a STABLE
nearby element (status label, header text, etc.) is sense-able.
Resolution: adapter finds anchor via host-resolve → anchor center
in normalized [0, 1] viewport coords → add (dx, dy) shift (also
normalized — dx: 0.15 = 15% of screen width to the right) → clamp
to [0, 1] → IOHID synthesize via tap_at_norm_coord.
Compared to Selector::Anchor (spatial-relation chain — “find a
node near/below/leftOf <anchor>”), AnchorRelative is an escape
hatch family (§9 #3 sibling to tap_at_coord) — the target itself
has no a11y form, only the anchor does. Adapter dispatches directly
(find_norm_coord(anchor) → tap_at_coord), bypassing resolver.
Fields
anchor: Box<Selector>Anchor selector — any other Selector form (Id / Text / Role / LocalizedText / etc.). Must resolve to exactly one node; adapter uses centroid.
Point
{ point: [0.5, 0.9] } — v5.20 c2 (a11y-i18n initiative L7 layer,
last-resort within fallback chain). Direct viewport-normalized
coord. Equivalent to Step::TapAtPoint for standalone use, but
expressed as a Selector for uniform composition inside
Selector::Fallback { chain: Vec<Selector> }. Adapter dispatches
directly via tap_at_coord; never reaches resolver.
Fallback
{ fallback: [<selector1>, <selector2>, ...] } — v5.20 c2 (a11y-i18n
initiative L7 layer). Sequential resolution chain — adapter tries
each sub-selector in order; first hit wins. Misses are recorded
for AI-readable error report (sense_layer / missing_id_hint /
suggested_fixes). On all-miss, surfaces ElementNotFound with a
chain-trace hint listing every attempted layer.
Composes L1-L7 into a single tapOn — typical pattern:
tapOn:
fallback:
- { id: "submit-btn" } # L1
- { localized_text: { en: "Submit", ja: "送信" } } # L4
- { ocrText: "Submit" } # L5
- { anchored: { anchor: { id: "form-area" }, dx: 0.0, dy: 0.05 } } # L6
- { point: [0.5, 0.9] } # L7Adapter dispatches each sub-selector through the standard run_tap path; chain element types are limited only by what run_tap can dispatch (any concrete Selector variant). Nesting Fallback inside Fallback is allowed but discouraged (no defined precedence beyond outer-then-inner).