# Specification: Browser Automation Library
## Vision
Browsing is a **standalone browser automation library** built on Chrome DevTools Protocol (CDP). It provides programmatic control over Chrome/Chromium for navigation, element interaction, content extraction, and screenshot capture — without any dependency on LLMs or AI services.
The core insight: **CDP is an engine. Browsing is the control panel.**
Most headless browsing tools are thin wrappers around CDP. They expose the protocol with a slightly nicer API but leave all the hard problems — interpreting pages, finding elements reliably, handling dynamic SPAs — to the developer. Browsing solves these problems directly with semantic DOM understanding and resilient element interaction.
## Design Principles
1. **CDP is invisible** — Users think about tasks, not WebSocket messages or `backend_node_id`
2. **Semantic over syntactic** — Understand what a page means, not just what its DOM contains
3. **Resilient over brittle** — Self-healing element resolution, not fragile selectors
4. **Stateful over stateless** — Session context and memory, not independent commands
5. **Observable over opaque** — DOM state snapshots and action traces for debugging
## Architecture Layers
### Layer 1: Perception — Semantic DOM Understanding
**Problem**: CDP gives you a raw DOM tree. Determining "this is a search form" from `<div class="x3f9"><input ...></div>` requires interpretation.
**Solution**: Enrich the DOM with inferred semantics before serialization.
- **Semantic role inference**: Classify elements by what they DO, not just what they ARE
- `SearchForm`, `LoginForm`, `Navigation`, `ProductCard`, `Article`, `Pagination`, `FilterPanel`
- **Page intent classification**: On load, classify the page type
- Login page, search results, product listing, product detail, article, error page, CAPTCHA
- **Form schema extraction**: Auto-detect form purpose and field metadata
- "This is a search form with a query field and a category dropdown"
- **Action affordance tagging**: For each interactive element, tag what action it enables
- "Submit form 'search'", "Navigate to /products", "Toggle filter panel"
- **Content hierarchy**: Distinguish main content, navigation, sidebar, ads, footer
**Implementation**: Extend `EnhancedDOMTreeNode` and `DOMInteractedElement` with `semantic_role` and `affordance` fields. Add a `PerceptionEngine` that runs as a post-processing pass after DOM tree construction.
**Success criteria**: DOM output includes semantic annotations that reduce ambiguity, without significantly increasing output size (achieved by pruning non-semantic elements).
### Layer 2: Action — Resilient Element Interaction
**Problem**: CDP relies on `backend_node_id` and XPath selectors that break constantly on React/Vue/Angular SPAs. Element indices shift, IDs are dynamic, and class names are hashed.
**Solution**: Multi-factor element resolution with automatic fallback.
- **Resolution chain**: When interacting with an element:
1. **Index resolver** — Look up by index in `selector_map` (current approach)
2. **Text match resolver** — Search by text content if index is stale
3. **Semantic resolver** — "Find the submit button in the login form"
4. **JavaScript resolver** — Execute JS to find by text/role/attribute
- **Self-healing selectors**: When an element is not found, automatically attempt the next resolver in the chain
- **Multi-factor addressing**: Combine text, position, visual features, and DOM path for robust identification
**Implementation**: Define an `ElementResolver` trait. Implement each strategy as a separate resolver. Chain them in a `FallbackResolver`.
**Success criteria**: Automation succeeds on dynamic SPAs where element indices shift between steps, with zero manual intervention.
### Layer 3: Memory — Persistent Session Context
**Problem**: Every session starts from zero. No memory of past site interactions, quirks, or progress.
**Solution**: Persistent memory layer.
- **Site-specific strategies**: Learn and remember how to interact with specific sites
- "On example-airline.com, the search button is inside a shadow DOM"
- **Credential profiles**: Recognize login pages and auto-fill from known profiles
- **Progress tracking**: For long-running tasks, track what's been done and what's remaining
- "Checked pages 1-3 of results, need to check page 4"
- **Session persistence**: Save and resume long-running tasks
**Implementation**: `SessionMemory` trait with a default `FileSystemMemory` implementation. Hook memory read/write into the session lifecycle.
**Success criteria**: Second visit to a site is measurably faster and more reliable than the first.
### Layer 4: Observability — Session Recording & Tracing
**Problem**: When automation fails, debugging means reading protocol logs. There's no easy way to "see" what the browser saw.
**Solution**: Capture DOM state and action traces for debugging.
- **DOM snapshots**: Capture serialized DOM state at each step
- **Action traces**: Record actions taken with element context and results
- **Screenshots** (optional): Capture visual state for visual debugging
**Success criteria**: When automation fails, a developer can inspect the DOM snapshot and action trace to understand the failure in under 2 minutes.
## Architecture
```
src/
├── perception/ # Semantic DOM enrichment, page classification
│ ├── engine.rs # PerceptionEngine — runs post-DOM-build
│ ├── mod.rs # Module exports
│ └── engine_test.rs # Perception tests
├── browser/ # Browser lifecycle and CDP communication
│ ├── session.rs # Browser session (BrowserClient impl)
│ ├── tab_manager.rs # Tab operations
│ ├── navigation.rs # Navigation operations
│ ├── screenshot.rs # Screenshot operations
│ ├── cdp.rs # CDP WebSocket client
│ ├── launcher.rs # Browser launcher
│ ├── profile.rs # Browser configuration
│ ├── auth.rs # HTTP auth / proxy / extra headers
│ ├── console.rs # Console log capture
│ ├── cookies.rs # Cookie management
│ ├── coverage.rs # JS/CSS coverage tracking
│ ├── dialog.rs # Dialog handling (alert/confirm/prompt)
│ ├── download.rs # Download monitoring
│ ├── filechooser.rs # File chooser interception
│ ├── har.rs # Network traffic HAR capture
│ ├── interceptor.rs # Network request interception
│ ├── pdf.rs # PDF export
│ ├── performance.rs # Performance metrics
│ ├── permissions.rs # Browser permissions API
│ ├── serviceworker.rs # Service worker management
│ ├── touch.rs # Touch gesture emulation
│ ├── tracing.rs # Chrome tracing / CPU profiling
│ ├── viewport.rs # Viewport / device emulation
│ ├── worker.rs # Web worker management
│ ├── views.rs # Data types
│ └── mod.rs
├── dom/ # DOM processing
│ ├── processor.rs # DOMProcessor trait impl
│ ├── serializer.rs # Text serialization
│ ├── service.rs # DOM service
│ ├── tree_builder.rs # DOM tree construction
│ ├── cdp_client.rs # CDP wrapper for DOM
│ ├── html_converter.rs # HTML to markdown
│ ├── views.rs # Data types
│ └── mod.rs
├── tools/ # Action system
│ ├── service.rs # Tools registry
│ ├── handlers/ # Action handlers
│ │ ├── navigation.rs
│ │ ├── interaction.rs
│ │ ├── tabs.rs
│ │ ├── content.rs
│ │ ├── advanced.rs
│ │ └── mod.rs
│ ├── views.rs # Data types
│ └── mod.rs
├── traits/ # Core trait abstractions
│ ├── browser_client.rs # BrowserClient trait
│ ├── dom_processor.rs # DOMProcessor trait
│ └── mod.rs
├── actor/ # Low-level interactions
│ ├── page.rs # Page operations
│ ├── element.rs # Element operations
│ ├── mouse.rs # Mouse interactions
│ ├── keyboard.rs # Keyboard input
│ └── mod.rs
├── config.rs # Configuration
├── error.rs # Error types
├── logging.rs # Logging setup
├── metrics.rs # Metrics collection
├── utils.rs # Utilities
├── views.rs # Shared data types
└── lib.rs # Public API
```
## Browser Feature Managers
The `browser/` module provides 20+ managers that wrap CDP domains into ergonomic, testable Rust APIs. Each manager follows the same pattern: `new(cdp_client) → async methods → event subscription via CdpClient::subscribe_events`.
### Dialog & File Chooser
| `DialogManager` | `Page` | `enable`, `accept`, `dismiss`, `get_pending_dialogs`, `set_auto_dismiss` | `DialogType`, `DialogInfo` |
| `FileChooserManager` | `Page` + `DOM` | `enable`, `disable`, `set_files`, `get_pending` | `FileChooserInfo` |
### Device & Viewport Emulation
| `ViewportManager` | `Emulation` + `Page` | `set_device_metrics`, `set_user_agent`, `set_timezone`, `set_media_features`, `set_color_scheme`, `set_vision_deficiency`, `set_locale`, `override_languages_and_fonts`, `preset_device` | `DevicePreset`, `DeviceMetrics`, `ViewportConfig`, `ColorScheme`, `MediaFeature`, `VisionDeficiency` |
### Permissions & Security
| `PermissionsManager` | `Browser` | `grant`, `reset`, `grant_camera`, `grant_microphone`, `grant_all_common` | `PermissionType` (16 variants) |
| `AuthManager` | `Network` | `set_basic_auth`, `set_proxy_auth`, `clear_auth` | `AuthCredentials` |
### Network & Performance
| `HarManager` | `Network` | `start_capture`, `export_har`, `export_json`, `entry_count`, `clear` | `HarLog`, `HarEntry`, `HarRequest`, `HarResponse`, `HarHeader` |
| `NetworkInterceptor` | `Fetch` | `enable(patterns)`, `disable`, `continue_request`, `fulfill_request`, `fail_request` | `InterceptPattern`, `InterceptedRequest`, `MockResponse` |
| `PerformanceManager` | `Performance` | `get_metrics`, `get_metric_values`, `get_heap_usage` | `PerformanceMetrics` |
| `CoverageManager` | `Profiler` + `CSS` | `start_js_coverage`, `stop_js_coverage`, `start_css_coverage`, `stop_css_coverage` | `CoverageResult`, `CoverageRange` |
| `TracingManager` | `Tracing` | `start(config)`, `end()`, `export_trace(path, result)`, `event_count()` | `TraceConfig`, `TraceResult` |
### Touch & Workers
| `TouchManager` | `Input` | `dispatch_touch`, `tap`, `double_tap`, `swipe`, `pinch`, `long_press` | `TouchPoint` |
| `WorkerManager` | `Target` + `Runtime` | `start_discovery`, `attach`, `detach`, `evaluate_in_worker`, `list_workers` | `WorkerInfo` |
| `ServiceWorkerManager` | `ServiceWorker` | `enable`, `disable`, `stop_all`, `start`, `unregister`, `update` | `ServiceWorkerRegistration`, `ServiceWorkerVersion` |
### Raw CDP Access
| `Browser::raw_cdp_command(method, params)` | Sends any CDP command directly and returns the raw JSON response |
| `BrowserClient::raw_cdp_command(method, params)` | Trait method providing the same power-user access |
### Wiring
All managers are accessible from `Browser` via `browser.manager_name()` and from any `BrowserClient` implementation via the default trait method `fn manager_name(&self) -> Result<Manager>`. They are also re-exported from `lib.rs` for direct use.
## Phased Implementation
### Phase 1: Perception (Complete)
- Add `semantic_role` to `EnhancedDOMTreeNode` and `DOMInteractedElement`
- Implement `PerceptionEngine` with heuristic-based role inference
- Add page intent classification to DOM service
- Update serializer to include semantic annotations
- **Verify**: DOM output includes semantic tags; output size does not increase significantly
### Phase 2: Resilient Action (Complete)
- [x] Define `ElementResolver` with `ResolverConfig` and `ResolutionStrategy`
- [x] Implement 5-strategy fallback chain: Index → Selector → TextMatch → SemanticRole → JavaScript
- [x] Build `ElementResolver::resolve` with confidence scoring
- [x] Update action handlers to use resolver chain (`InteractionHandler`)
- [x] Add unit tests for resolver config, strategies, and selector map
- [x] **Verify**: `_verify_backend_node` uses real CDP `DOM.describeNode` instead of stub
### Phase 3: Memory (Complete)
- [x] Implement `AgentMemory` with `ProgressTracker`, `CredentialProfile`, `SiteStrategy`
- [x] Add `save_to_file` / `load_from_file` (excludes credentials for security)
- [x] Build `build_context` for domain-aware prompt enrichment
- [x] Wire `AgentMemory` into `ActionContext` so action handlers can access memory
- [x] **Verify**: Memory persistence round-trip works (progress + strategies saved/loaded)
### Phase 4: Observability (In Progress)
- [x] `SessionRecorder` — Records steps with timestamp, action type, params, URL, result, duration
- [x] JSON trace export — `SessionRecorder::export_to_file` writes `SessionTrace` with metadata
- [x] Configurable recording — `RecorderConfig` controls max steps, DOM capture, truncation
- [x] Automatic action recording — `Tools::act` records every action when a recorder is provided
- [ ] DOM snapshot capture at each step (pass via `dom_snapshot` param to `Tools::act`)
- [ ] **Verify**: A failed session produces a trace that explains the failure
## Success Metrics
| Element resolution success on dynamic SPAs | >95% |
| Multi-step task completion without intervention | >85% |
| Time to debug a failed run | <2 minutes |
| Second-visit speedup vs first visit | >30% |
## Risk & Mitigation
| Output size increase from semantic annotations | Prune non-semantic elements aggressively; use hierarchical summarization |
| Complexity explosion from new modules | Implement one module at a time; each module is optional and can be disabled |
| Heuristic-based inference being wrong | Confidence scoring; fall back to raw DOM when confidence is low |
| Performance overhead of perception layer | Run perception asynchronously; cache results per page load |
