# Specification: Browser Automation Library
## Vision
Browsing is a **standalone browser automation library** built on Chrome DevTools Protocol (CDP). It provides programmatic control over Chrome/Chromium for navigation, element interaction, content extraction, and screenshot capture — without any dependency on LLMs or AI services.
The core insight: **CDP is an engine. Browsing is the control panel.**
Most headless browsing tools are thin wrappers around CDP. They expose the protocol with a slightly nicer API but leave all the hard problems — interpreting pages, finding elements reliably, handling dynamic SPAs — to the developer. Browsing solves these problems directly with semantic DOM understanding and resilient element interaction.
## Design Principles
1. **CDP is invisible** — Users think about tasks, not WebSocket messages or `backend_node_id`
2. **Semantic over syntactic** — Understand what a page means, not just what its DOM contains
3. **Resilient over brittle** — Self-healing element resolution, not fragile selectors
4. **Stateful over stateless** — Session context and memory, not independent commands
5. **Observable over opaque** — DOM state snapshots and action traces for debugging
## Architecture Layers
### Layer 1: Perception — Semantic DOM Understanding
**Problem**: CDP gives you a raw DOM tree. Determining "this is a search form" from `<div class="x3f9"><input ...></div>` requires interpretation.
**Solution**: Enrich the DOM with inferred semantics before serialization.
- **Semantic role inference**: Classify elements by what they DO, not just what they ARE
- `SearchForm`, `LoginForm`, `Navigation`, `ProductCard`, `Article`, `Pagination`, `FilterPanel`
- **Page intent classification**: On load, classify the page type
- Login page, search results, product listing, product detail, article, error page, CAPTCHA
- **Form schema extraction**: Auto-detect form purpose and field metadata
- "This is a search form with a query field and a category dropdown"
- **Action affordance tagging**: For each interactive element, tag what action it enables
- "Submit form 'search'", "Navigate to /products", "Toggle filter panel"
- **Content hierarchy**: Distinguish main content, navigation, sidebar, ads, footer
**Implementation**: Extend `EnhancedDOMTreeNode` and `DOMInteractedElement` with `semantic_role` and `affordance` fields. Add a `PerceptionEngine` that runs as a post-processing pass after DOM tree construction.
**Success criteria**: DOM output includes semantic annotations that reduce ambiguity, without significantly increasing output size (achieved by pruning non-semantic elements).
### Layer 2: Action — Resilient Element Interaction
**Problem**: CDP relies on `backend_node_id` and XPath selectors that break constantly on React/Vue/Angular SPAs. Element indices shift, IDs are dynamic, and class names are hashed.
**Solution**: Multi-factor element resolution with automatic fallback.
- **Resolution chain**: When interacting with an element:
1. **Index resolver** — Look up by index in `selector_map` (current approach)
2. **Text match resolver** — Search by text content if index is stale
3. **Semantic resolver** — "Find the submit button in the login form"
4. **JavaScript resolver** — Execute JS to find by text/role/attribute
- **Self-healing selectors**: When an element is not found, automatically attempt the next resolver in the chain
- **Multi-factor addressing**: Combine text, position, visual features, and DOM path for robust identification
**Implementation**: Define an `ElementResolver` trait. Implement each strategy as a separate resolver. Chain them in a `FallbackResolver`.
**Success criteria**: Automation succeeds on dynamic SPAs where element indices shift between steps, with zero manual intervention.
### Layer 3: Memory — Persistent Session Context
**Problem**: Every session starts from zero. No memory of past site interactions, quirks, or progress.
**Solution**: Persistent memory layer.
- **Site-specific strategies**: Learn and remember how to interact with specific sites
- "On example-airline.com, the search button is inside a shadow DOM"
- **Credential profiles**: Recognize login pages and auto-fill from known profiles
- **Progress tracking**: For long-running tasks, track what's been done and what's remaining
- "Checked pages 1-3 of results, need to check page 4"
- **Session persistence**: Save and resume long-running tasks
**Implementation**: `SessionMemory` trait with a default `FileSystemMemory` implementation. Hook memory read/write into the session lifecycle.
**Success criteria**: Second visit to a site is measurably faster and more reliable than the first.
### Layer 4: Observability — Session Recording & Tracing
**Problem**: When automation fails, debugging means reading protocol logs. There's no easy way to "see" what the browser saw.
**Solution**: Capture DOM state and action traces for debugging.
- **DOM snapshots**: Capture serialized DOM state at each step
- **Action traces**: Record actions taken with element context and results
- **Screenshots** (optional): Capture visual state for visual debugging
**Success criteria**: When automation fails, a developer can inspect the DOM snapshot and action trace to understand the failure in under 2 minutes.
## Architecture
```
src/
├── perception/ # Semantic DOM enrichment, page classification
│ ├── engine.rs # PerceptionEngine — runs post-DOM-build
│ ├── mod.rs # Module exports
│ └── engine_test.rs # Perception tests
├── browser/ # Browser lifecycle and CDP communication
│ ├── session.rs # Browser session (BrowserClient impl)
│ ├── tab_manager.rs # Tab operations
│ ├── navigation.rs # Navigation operations
│ ├── screenshot.rs # Screenshot operations
│ ├── cdp.rs # CDP WebSocket client
│ ├── launcher.rs # Browser launcher
│ ├── profile.rs # Browser configuration
│ ├── views.rs # Data types
│ └── mod.rs
├── dom/ # DOM processing
│ ├── processor.rs # DOMProcessor trait impl
│ ├── serializer.rs # Text serialization
│ ├── service.rs # DOM service
│ ├── tree_builder.rs # DOM tree construction
│ ├── cdp_client.rs # CDP wrapper for DOM
│ ├── html_converter.rs # HTML to markdown
│ ├── views.rs # Data types
│ └── mod.rs
├── tools/ # Action system
│ ├── service.rs # Tools registry
│ ├── handlers/ # Action handlers
│ │ ├── navigation.rs
│ │ ├── interaction.rs
│ │ ├── tabs.rs
│ │ ├── content.rs
│ │ ├── advanced.rs
│ │ └── mod.rs
│ ├── views.rs # Data types
│ └── mod.rs
├── traits/ # Core trait abstractions
│ ├── browser_client.rs # BrowserClient trait
│ ├── dom_processor.rs # DOMProcessor trait
│ └── mod.rs
├── actor/ # Low-level interactions
│ ├── page.rs # Page operations
│ ├── element.rs # Element operations
│ ├── mouse.rs # Mouse interactions
│ ├── keyboard.rs # Keyboard input
│ └── mod.rs
├── config.rs # Configuration
├── error.rs # Error types
├── logging.rs # Logging setup
├── metrics.rs # Metrics collection
├── utils.rs # Utilities
├── views.rs # Shared data types
└── lib.rs # Public API
```
## Phased Implementation
### Phase 1: Perception (Complete)
- Add `semantic_role` to `EnhancedDOMTreeNode` and `DOMInteractedElement`
- Implement `PerceptionEngine` with heuristic-based role inference
- Add page intent classification to DOM service
- Update serializer to include semantic annotations
- **Verify**: DOM output includes semantic tags; output size does not increase significantly
### Phase 2: Resilient Action (Complete)
- [x] Define `ElementResolver` with `ResolverConfig` and `ResolutionStrategy`
- [x] Implement 5-strategy fallback chain: Index → Selector → TextMatch → SemanticRole → JavaScript
- [x] Build `ElementResolver::resolve` with confidence scoring
- [x] Update action handlers to use resolver chain (`InteractionHandler`)
- [x] Add unit tests for resolver config, strategies, and selector map
- [x] **Verify**: `_verify_backend_node` uses real CDP `DOM.describeNode` instead of stub
### Phase 3: Memory (Complete)
- [x] Implement `AgentMemory` with `ProgressTracker`, `CredentialProfile`, `SiteStrategy`
- [x] Add `save_to_file` / `load_from_file` (excludes credentials for security)
- [x] Build `build_context` for domain-aware prompt enrichment
- [x] Wire `AgentMemory` into `ActionContext` so action handlers can access memory
- [x] **Verify**: Memory persistence round-trip works (progress + strategies saved/loaded)
### Phase 4: Observability (In Progress)
- [x] `SessionRecorder` — Records steps with timestamp, action type, params, URL, result, duration
- [x] JSON trace export — `SessionRecorder::export_to_file` writes `SessionTrace` with metadata
- [x] Configurable recording — `RecorderConfig` controls max steps, DOM capture, truncation
- [x] Automatic action recording — `Tools::act` records every action when a recorder is provided
- [ ] DOM snapshot capture at each step (pass via `dom_snapshot` param to `Tools::act`)
- [ ] **Verify**: A failed session produces a trace that explains the failure
## Success Metrics
| Element resolution success on dynamic SPAs | >95% |
| Multi-step task completion without intervention | >85% |
| Time to debug a failed run | <2 minutes |
| Second-visit speedup vs first visit | >30% |
## Risk & Mitigation
| Output size increase from semantic annotations | Prune non-semantic elements aggressively; use hierarchical summarization |
| Complexity explosion from new modules | Implement one module at a time; each module is optional and can be disabled |
| Heuristic-based inference being wrong | Confidence scoring; fall back to raw DOM when confidence is low |
| Performance overhead of perception layer | Run perception asynchronously; cache results per page load |
