# Browsing
**Lightweight, headless-first MCP/API for browser automation**
A concise MCP server and Rust library: **navigate**, **get_links**, **follow_link**, **list_content** (links+images), **get_content**, **get_image**, **save_content**, **screenshot** (full or element). Lazy browser init. Parallel reads via RwLock. Runs headless by default with 17+ lightweight Chrome flags for fast, minimal operation. No LLM or AI dependencies.
## ๐ฏ Usage Modes
1. **๐ MCP Server** (primary) - `navigate`, `get_links`, `follow_link`, `list_content`, `get_content`, `get_image`, `save_content`, `screenshot`, `generate_sitemap` tools
2. **โจ๏ธ CLI** - Browser automation tasks
3. **๐ฆ Library** - Browser automation via CDP
## โจ Why Browsing?
Browser automation is challenging. You need to:
- **Extract structured data from unstructured HTML** - Parse complex DOM trees into usable representations
- **Handle browser automation reliably** - Manage browser lifecycle, CDP connections, and process management
- **Maintain testability** - Mock components for unit testing without real browsers
- **Support extensibility** - Add custom actions and browser backends
**Browsing solves all of this** with a clean, modular, and well-tested architecture.
## ๐ง Why Better Than Chrome DevTools?
Chrome DevTools Protocol (CDP) is an **engine** โ it gives you raw browser control, but you must build all intelligence yourself. Most "headless browsing" tools are just CDP wrappers with slightly nicer APIs.
Browsing is a **self-driving car** on top of that engine. It adds five intelligence layers that are hard to replicate by simply wrapping CDP:
| **Perception** | Raw DOM tree | Semantic understanding: page intent, form schemas, action affordances |
| **Action** | Brittle `backend_node_id`/XPath | Self-healing element resolution: index โ text โ semantic fallback |
| **Memory** | Nothing persists | Session history, site-specific strategies |
| **Observability** | Protocol logs | DOM state snapshots and action traces |
**The goal**: CDP becomes invisible. Users think about tasks, not WebSocket messages or element IDs.
## ๐ฏ Key Features
### ๐๏ธ Trait-Based Architecture
- **BrowserClient trait** - Abstract browser operations for easy mocking and alternative backends
- **DOMProcessor trait** - Pluggable DOM processing implementations
- **ActionHandler trait** - Extensible action system for custom behaviors
### ๐ Full Browser Automation
- **Headless by default** โ `BrowserProfile::default()` runs headless with `--headless=new`
- **17+ lightweight flags** โ disables extensions, sync, background networking, logging, and more
- Cross-platform support (macOS, Linux, Windows)
- Automatic browser detection
- Chrome DevTools Protocol (CDP) integration
- Tab management (create, switch, close)
- Screenshot capture (page and element-level)
### ๐ Advanced DOM Processing
- Full CDP integration (DOM, AX tree, Snapshot)
- Text serialization with interactive element indices
- Accessibility tree support for better semantic understanding
### ๐ง Extensible & Maintainable
- Manager-based architecture (TabManager, NavigationManager, ScreenshotManager)
- Custom action registration
- Utility traits for reduced code duplication
- Comprehensive test coverage (350+ tests)
## ๐ฆ Installation
### As a Library
```toml
[dependencies]
browsing = "0.1"
tokio = { version = "1.40", features = ["full"] }
```
### As a CLI Tool
```bash
cargo install --path . --bin browsing
```
### As an MCP Server
```bash
cargo build --release --bin browsing-mcp
```
## ๐ ๏ฟฝ Quick Start## ๏ฟฝ Quick Start
### 1๏ธโฃ CLI Usage
```bash
# Run a browser automation task (headless by default)
browsing run "Navigate to example.com and extract content" --url https://news.ycombinator.com
# Run with visible browser
browsing run "Navigate to example.com and extract content" --url https://news.ycombinator.com --no-headless
# Launch a headless browser and get CDP URL (headless by default)
browsing launch
# Launch with visible browser
browsing launch --no-headless
# Connect to existing browser
browsing connect ws://localhost:9222/devtools/browser/abc123
```
**๐ [Full CLI Documentation](docs/CLI_USAGE.md)**
### 2๏ธโฃ MCP Server Usage
Configure in Claude Desktop (`~/Library/Application Support/Claude/claude_desktop_config.json`):
```json
{
"mcpServers": {
"browsing": {
"command": "/path/to/browsing/target/release/browsing-mcp",
"env": {
"BROWSER_USE_HEADLESS": "true"
}
}
}
}
```
Then ask Claude:
```
"Navigate to rust-lang.org, get the links, follow the second link, and screenshot the main content area"
```
**๐ [Full MCP Documentation](docs/MCP_USAGE.md)**
### 3๏ธโฃ Library Usage
```rust
use anyhow::Result;
use browsing::Browser;
use browsing::browser::BrowserProfile;
#[tokio::main]
async fn main() -> Result<()> {
browsing::init();
let mut browser = Browser::new(BrowserProfile::default());
browser.start().await?;
browser.navigate("https://example.com").await?;
let title = browser.get_current_page_title().await?;
println!("Title: {}", title);
let _ = browser.stop().await;
Ok(())
}
```
**๐ [Full Library Documentation](docs/LIBRARY_USAGE.md)**
### Browser Launch Options
```rust
use browsing::{Browser, BrowserProfile};
// Option 1: Auto-launch headless browser (default)
let profile = BrowserProfile::default(); // headless = true by default
let browser = Browser::new(profile);
// Option 2: Launch with visible browser
let profile = BrowserProfile::default().with_headless(false);
let browser = Browser::new(profile);
// Option 3: Connect to existing browser
let browser = Browser::new(BrowserProfile::default())
.with_cdp_url("http://localhost:9222".to_string());
// Option 4: Custom browser executable
use browsing::browser::launcher::BrowserLauncher;
let launcher = BrowserLauncher::new(profile)
.with_executable_path(std::path::PathBuf::from("/path/to/chrome"));
```
### Using Traits for Testing
#### Standalone Browser Testing (no AI required)
```rust
use browsing::traits::{BrowserClient, DOMProcessor};
use std::sync::Arc;
// Test your browser automation logic directly
```
## ๐ Usage Examples
### Content Download
```rust
use browsing::{Browser, BrowserProfile};
use browsing::dom::DOMProcessorImpl;
use browsing::traits::DOMProcessor;
#[tokio::main]
async fn main() -> browsing::error::Result<()> {
let mut browser = Browser::new(BrowserProfile::default());
browser.start().await?;
// Navigate to website
browser.navigate("https://www.ibm.com").await?;
tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
// Extract content
let cdp_client = browser.get_cdp_client()?;
let session_id = browser.get_session_id()?;
let target_id = browser.get_current_target_id()?;
let dom_processor = DOMProcessorImpl::new()
.with_cdp_client(cdp_client, session_id)
.with_target_id(target_id);
let state = dom_processor.get_serialized_dom().await?;
let page_content = state.text_representation(None).unwrap_or_default();
println!("Extracted {} bytes of content", page_content.len());
// Save to file
std::fs::write("ibm_content.txt", page_content)?;
Ok(())
}
```
**Run this example:**
```bash
cargo run --example ibm_content_download
```
### Screenshot Capture
```rust
use browsing::Browser;
let browser = Browser::new(BrowserProfile::default());
browser.start().await?;
// Full page screenshot
let screenshot_data = browser.take_screenshot(
Some("screenshot.png"), // path
true, // full_page
None, // format
None, // quality
).await?;
// Viewport only
let viewport = browser.take_screenshot(
Some("viewport.png"),
false,
None,
None,
).await?;
```
### Direct Browser Control
```rust
use browsing::{Browser, BrowserProfile};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut browser = Browser::new(BrowserProfile::default());
browser.start().await?;
// Navigate
browser.navigate("https://example.com").await?;
// Get current URL
let url = browser.get_current_url().await?;
println!("Current URL: {}", url);
// Tab management
use browsing::traits::BrowserClient;
browser.create_tab(Some("https://hackernews.com")).await?;
let tabs = browser.get_tabs().await?;
println!("Open tabs: {}", tabs.len());
// Switch tabs
browser.switch_to_tab(&tabs[0].target_id).await?;
Ok(())
}
```
### Custom Actions
```rust
use browsing::tools::views::{ActionHandler, ActionParams, ActionContext, ActionResult};
use browsing::error::Result;
use browsing::tools::Tools;
struct CustomActionHandler;
#[async_trait::async_trait]
impl ActionHandler for CustomActionHandler {
async fn execute(
&self,
params: &ActionParams,
context: &mut ActionContext<'_>,
) -> Result<ActionResult> {
// Custom action logic here
Ok(ActionResult {
extracted_content: Some("Custom result".to_string()),
..Default::default()
})
}
}
// Register custom action
let mut tools = Tools::default();
tools.register_custom_action(
"custom_action".to_string(),
"Description of custom action".to_string(),
None, // domains
CustomActionHandler,
);
```
## ๐๏ธ Architecture
Browsing follows **SOLID principles** with a focus on separation of concerns, testability, and maintainability.
```
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Browser Automation โ
โ โโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Browser โ DOMProcessor โ Tools โ โ
โ โ (trait) โ (trait) โ โ โ
โ โโโโโโโโฌโโโโโโโดโโโโโโโฌโโโโโโโโดโโโโโโโโโโโโโโโโโฌโโโโโโโโโ โ
โ โ โ โ โ
โโโโโโโโโโโผโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโ
โ โ โ
โโโโโโโผโโโโโโโ โโโโโผโโโโโ โโโโโโโโโโผโโโโโโ
โ Browser โ โDomSvc โ โ Handlers โ
โ โ โ โ โ โ
โTabManager โ โCDP โ โNavigation โ
โNavManager โ โHTML โ โInteraction โ
โScreenshot โ โTree โ โTabs โ
โ โ โBuilder โ โContent โ
โโโโโโโโโโโโโโ โโโโโโโโโโ โโโโโโโโโโโโโโโโ
```
### Key Components
| **Browser** | Manages browser session and lifecycle | Implements `BrowserClient` |
| **DOMProcessor** | Extracts and serializes DOM | Implements `DOMProcessor` |
| **Tools** | Action registry and execution | Uses `BrowserClient` trait |
| **Handlers** | Specific action implementations | Use `ActionHandler` trait |
## ๐ Project Structure
```
browsing/
โโโ src/
โ โโโ agent/ # Agent view types (used by tools layer)
โ โ โโโ views.rs # Data types
โ โ โโโ memory.rs # Session memory types
โ โโโ browser/ # Browser management
โ โ โโโ session.rs # Browser session (BrowserClient impl)
โ โ โโโ tab_manager.rs # Tab operations
โ โ โโโ navigation.rs # Navigation operations
โ โ โโโ screenshot.rs # Screenshot operations
โ โ โโโ cdp.rs # CDP WebSocket client
โ โ โโโ launcher.rs # Browser launcher
โ โ โโโ profile.rs # Browser configuration
โ โโโ dom/ # DOM processing
โ โ โโโ processor.rs # DOMProcessor trait impl
โ โ โโโ serializer.rs # Text serialization
โ โ โโโ tree_builder.rs # DOM tree construction
โ โ โโโ cdp_client.rs # CDP wrapper for DOM
โ โ โโโ html_converter.rs # HTML to markdown
โ โโโ tools/ # Action system
โ โ โโโ service.rs # Tools registry
โ โ โโโ handlers/ # Action handlers
โ โ โ โโโ navigation.rs
โ โ โ โโโ interaction.rs
โ โ โ โโโ tabs.rs
โ โ โ โโโ content.rs
โ โ โ โโโ advanced.rs
โ โ โโโ params.rs # Parameter extraction
โ โโโ traits/ # Core trait abstractions
โ โ โโโ browser_client.rs # BrowserClient trait
โ โ โโโ dom_processor.rs # DOMProcessor trait
โ โโโ actor/ # Low-level interactions
โ โ โโโ page.rs # Page operations
โ โ โโโ element.rs # Element operations
โ โ โโโ mouse.rs # Mouse interactions
โ โ โโโ keyboard.rs # Keyboard input
โ โโโ config.rs # Configuration
โ โโโ error.rs # Error types
โ โโโ logging.rs # Logging setup
โ โโโ metrics.rs # Metrics collection
โ โโโ utils.rs # Utilities
โ โโโ views.rs # Shared data types
โโโ Cargo.toml
```
## ๐จ Design Principles
### Trait-Facing Design
- **BrowserClient** - Abstract browser operations for testing and alternative backends
- **DOMProcessor** - Pluggable DOM processing implementations
- **ActionHandler** - Extensible action system
### Separation of Concerns
- **TabManager** - Tab operations (create, switch, close)
- **NavigationManager** - Navigation logic
- **ScreenshotManager** - Screenshot capture
- **Handlers** - Focused action implementations
### DRY (Don't Repeat Yourself)
- **ActionParams** - Reusable parameter extraction
- **JSONExtractor** - Centralized JSON parsing
- **SessionGuard** - Unified session access
### KISS (Keep It Simple, Stupid)
- Split complex methods into focused helpers
- Clear naming and single responsibility
- Minimal dependencies between modules
## ๐งช Testing
```bash
# Run all tests
cargo test
# Run with output
cargo test -- --nocapture
# Run specific test
cargo test test_browser_navigation
# Run integration tests only
cargo test --test integration
```
### Test Coverage
- **350+ tests** across all modules (all passing, 0 ignored)
- **Test files**:
- [actor_test.rs](tests/actor_test.rs) - Page, Element, Mouse, Keyboard operations
- [browser_lifecycle_test.rs](tests/browser_lifecycle_test.rs) - Browser start/stop lifecycle
- [browser_managers_test.rs](tests/browser_managers_test.rs) - Navigation, Screenshot, Tab managers
- [browser_test.rs](tests/browser_test.rs) - Browser integration
- [cdp_enhancements_test.rs](tests/cdp_enhancements_test.rs) - CDP client features
- [dom_extraction_test.rs](tests/dom_extraction_test.rs) - DOM extraction
- [dom_test.rs](tests/dom_test.rs) - DOM state and serialization
- [error_handling_test.rs](tests/error_handling_test.rs) - Error variants and propagation
- [error_recovery_test.rs](tests/error_recovery_test.rs) - Config validation and recovery
- [integration_test.rs](tests/integration_test.rs) - Action models, URL extraction, tools
- [integration_workflow_test.rs](tests/integration_workflow_test.rs) - Full workflow
- [security_test.rs](tests/security_test.rs) - Security validation
- [tools_handlers_test.rs](tests/tools_handlers_test.rs) - All action handlers
- [tools_test.rs](tests/tools_test.rs) - Tools registry
- [traits_test.rs](tests/traits_test.rs) - BrowserClient, DOMProcessor traits
- [utils_test.rs](tests/utils_test.rs) - URL extraction, signal handling
- [agent_test.rs](tests/agent_test.rs) - Agent view types
- [agent_service_test.rs](tests/agent_service_test.rs) - Agent state and history
- **Mock implementations** for deterministic testing
- **Trait-based mocking** for browser/DOM components
## โ ๏ธ Data Retention Policy
### Browser Data is NEVER Deleted
**IMPORTANT**: The `browsing` library **never deletes browser data** for safety reasons.
#### What This Means:
| **Bookmarks** | Never deleted |
| **History** | Never deleted |
| **Cookies** | Never deleted |
| **Passwords** | Never deleted |
| **Extensions** | Never deleted |
| **Cache** | Never deleted |
| **Temp Directories** | Never deleted (left in `/tmp/`) |
#### Why This Policy Exists:
1. **User Safety**: Users may specify a custom `user_data_dir` pointing to their real browser profile
2. **Catastrophe Prevention**: Accidentally deleting a user's real browser data (bookmarks, history, passwords) would be devastating
3. **Debugging**: Leaving temp directories allows inspection after crashes or failures
4. **User Control**: Users are responsible for managing their own browser data
#### How It Works:
When no `user_data_dir` is specified:
```rust
let profile = BrowserProfile {
user_data_dir: None, // Uses temp directory: /tmp/browser-use-1738369200000/
..Default::default()
};
```
When `browser.stop()` is called:
- โ
Browser process is killed
- โ
In-memory state is cleared
- โ User data directory is **NOT** deleted
#### Managing Temporary Data:
Users are responsible for cleanup:
```bash
# List browser temp directories
ls -la /tmp/browser-use-*
# Delete old temp directories (optional, manual cleanup)
rm -rf /tmp/browser-use-1738369200000/
```
#### Using a Custom Data Directory:
```rust
let profile = BrowserProfile {
user_data_dir: Some("/path/to/custom/profile".into()),
..Default::default()
};
```
**Warning**: If you point to your real browser profile, the library will NOT protect it. You're responsible for that directory.
## ๐ง Configuration
### Browser Profile
```rust
use browsing::BrowserProfile;
// Default: headless mode with lightweight flags
let profile = BrowserProfile::default();
// Explicit headless
let profile = BrowserProfile::default().with_headless(true);
// Visible browser for debugging
let profile = BrowserProfile::default().with_headless(false);
// With custom data directory
let profile = BrowserProfile {
user_data_dir: Some("/path/to/profile".into()),
..Default::default()
};
```
## ๐ API Documentation
Generate and view API docs:
```bash
cargo doc --open
```
