Browsing

Lightweight, headless-first MCP/API for browser automation

A concise MCP server and Rust library: navigate, get_links, follow_link, list_content (links+images), get_content, get_image, save_content, screenshot (full or element). Lazy browser init. Parallel reads via RwLock. Runs headless by default with 17+ lightweight Chrome flags for fast, minimal operation. No LLM or AI dependencies.

🎯 Usage Modes

1. 🔌 MCP Server (primary) - navigate, get_links, follow_link, list_content, get_content, get_image, save_content, screenshot, generate_sitemap tools
2. ⌨️ CLI - Browser automation tasks
3. 📦 Library - Browser automation via CDP

✨ Why Browsing?

Browser automation is challenging. You need to:

• Extract structured data from unstructured HTML - Parse complex DOM trees into usable representations
• Handle browser automation reliably - Manage browser lifecycle, CDP connections, and process management
• Maintain testability - Mock components for unit testing without real browsers
• Support extensibility - Add custom actions and browser backends

Browsing solves all of this with a clean, modular, and well-tested architecture.

🧠 Why Better Than Chrome DevTools?

Chrome DevTools Protocol (CDP) is an engine — it gives you raw browser control, but you must build all intelligence yourself. Most "headless browsing" tools are just CDP wrappers with slightly nicer APIs.

Browsing is a self-driving car on top of that engine. It adds five intelligence layers that are hard to replicate by simply wrapping CDP:

Layer	CDP (Raw Protocol)	Browsing (Browser Automation)
Perception	Raw DOM tree	Semantic understanding: page intent, form schemas, action affordances
Action	Brittle backend_node_id/XPath	Self-healing element resolution: index → text → semantic fallback
Memory	Nothing persists	Session history, site-specific strategies
Observability	Protocol logs	DOM state snapshots and action traces

The goal: CDP becomes invisible. Users think about tasks, not WebSocket messages or element IDs.

🎯 Key Features

🏗️ Trait-Based Architecture

• BrowserClient trait - Abstract browser operations for easy mocking and alternative backends
• DOMProcessor trait - Pluggable DOM processing implementations
• ActionHandler trait - Extensible action system for custom behaviors

🌐 Full Browser Automation

• Headless by default — BrowserProfile::default() runs headless with --headless=new
• 17+ lightweight flags — disables extensions, sync, background networking, logging, and more
• Cross-platform support (macOS, Linux, Windows)
• Automatic browser detection
• Chrome DevTools Protocol (CDP) integration
• Tab management (create, switch, close)
• Screenshot capture (page and element-level)

📊 Advanced DOM Processing

• Full CDP integration (DOM, AX tree, Snapshot)
• Text serialization with interactive element indices
• Accessibility tree support for better semantic understanding

🔧 Extensible & Maintainable

• Manager-based architecture (TabManager, NavigationManager, ScreenshotManager)
• Custom action registration
• Utility traits for reduced code duplication
• Comprehensive test coverage (350+ tests)

📦 Installation

As a Library

[dependencies]
browsing = "0.1"
tokio = { version = "1.40", features = ["full"] }

As a CLI Tool

cargo install --path . --bin browsing

As an MCP Server

cargo build --release --bin browsing-mcp

🚀 � Quick Start## � Quick Start

1️⃣ CLI Usage

# Run a browser automation task (headless by default)
browsing run "Navigate to example.com and extract content" --url https://news.ycombinator.com

# Run with visible browser
browsing run "Navigate to example.com and extract content" --url https://news.ycombinator.com --no-headless

# Launch a headless browser and get CDP URL (headless by default)
browsing launch

# Launch with visible browser
browsing launch --no-headless

# Connect to existing browser
browsing connect ws://localhost:9222/devtools/browser/abc123

📖 Full CLI Documentation

2️⃣ MCP Server Usage

Configure in Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "browsing": {
      "command": "/path/to/browsing/target/release/browsing-mcp",
      "env": {
        "BROWSER_USE_HEADLESS": "true"
      }
    }
  }
}

Then ask Claude:

"Navigate to rust-lang.org, get the links, follow the second link, and screenshot the main content area"

📖 Full MCP Documentation

3️⃣ Library Usage

use anyhow::Result;
use browsing::Browser;
use browsing::browser::BrowserProfile;

#[tokio::main]
async fn main() -> Result<()> {
    browsing::init();

    let mut browser = Browser::new(BrowserProfile::default());
    browser.start().await?;

    browser.navigate("https://example.com").await?;

    let title = browser.get_current_page_title().await?;
    println!("Title: {}", title);

    let _ = browser.stop().await;
    Ok(())
}

📖 Full Library Documentation

Browser Launch Options

use browsing::{Browser, BrowserProfile};

// Option 1: Auto-launch headless browser (default)
let profile = BrowserProfile::default(); // headless = true by default
let browser = Browser::new(profile);

// Option 2: Launch with visible browser
let profile = BrowserProfile::default().with_headless(false);
let browser = Browser::new(profile);

// Option 3: Connect to existing browser
let browser = Browser::new(BrowserProfile::default())
    .with_cdp_url("http://localhost:9222".to_string());

// Option 4: Custom browser executable
use browsing::browser::launcher::BrowserLauncher;
let launcher = BrowserLauncher::new(profile)
    .with_executable_path(std::path::PathBuf::from("/path/to/chrome"));

Using Traits for Testing

Standalone Browser Testing (no AI required)

use browsing::traits::{BrowserClient, DOMProcessor};
use std::sync::Arc;

// Test your browser automation logic directly

📚 Usage Examples

Content Download

use browsing::{Browser, BrowserProfile};
use browsing::dom::DOMProcessorImpl;
use browsing::traits::DOMProcessor;

#[tokio::main]
async fn main() -> browsing::error::Result<()> {
    let mut browser = Browser::new(BrowserProfile::default());
    browser.start().await?;

    // Navigate to website
    browser.navigate("https://www.ibm.com").await?;
    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;

    // Extract content
    let cdp_client = browser.get_cdp_client()?;
    let session_id = browser.get_session_id()?;
    let target_id = browser.get_current_target_id()?;

    let dom_processor = DOMProcessorImpl::new()
        .with_cdp_client(cdp_client, session_id)
        .with_target_id(target_id);

    let state = dom_processor.get_serialized_dom().await?;
    let page_content = state.text_representation(None).unwrap_or_default();
    println!("Extracted {} bytes of content", page_content.len());

    // Save to file
    std::fs::write("ibm_content.txt", page_content)?;
    Ok(())
}

Run this example:

cargo run --example ibm_content_download

Screenshot Capture

use browsing::Browser;

let browser = Browser::new(BrowserProfile::default());
browser.start().await?;

// Full page screenshot
let screenshot_data = browser.take_screenshot(
    Some("screenshot.png"),  // path
    true,                     // full_page
    None,                     // format
    None,                     // quality
).await?;

// Viewport only
let viewport = browser.take_screenshot(
    Some("viewport.png"),
    false,
    None,
    None,
).await?;

Direct Browser Control

use browsing::{Browser, BrowserProfile};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut browser = Browser::new(BrowserProfile::default());
    browser.start().await?;

    // Navigate
    browser.navigate("https://example.com").await?;

    // Get current URL
    let url = browser.get_current_url().await?;
    println!("Current URL: {}", url);

    // Tab management
    use browsing::traits::BrowserClient;
    browser.create_tab(Some("https://hackernews.com")).await?;
    let tabs = browser.get_tabs().await?;
    println!("Open tabs: {}", tabs.len());

    // Switch tabs
    browser.switch_to_tab(&tabs[0].target_id).await?;

    Ok(())
}

Custom Actions

use browsing::tools::views::{ActionHandler, ActionParams, ActionContext, ActionResult};
use browsing::error::Result;
use browsing::tools::Tools;

struct CustomActionHandler;

#[async_trait::async_trait]
impl ActionHandler for CustomActionHandler {
    async fn execute(
        &self,
        params: &ActionParams,
        context: &mut ActionContext<'_>,
    ) -> Result<ActionResult> {
        // Custom action logic here
        Ok(ActionResult {
            extracted_content: Some("Custom result".to_string()),
            ..Default::default()
        })
    }
}

// Register custom action
let mut tools = Tools::default();
tools.register_custom_action(
    "custom_action".to_string(),
    "Description of custom action".to_string(),
    None,  // domains
    CustomActionHandler,
);

🏗️ Architecture

Browsing follows SOLID principles with a focus on separation of concerns, testability, and maintainability.

┌─────────────────────────────────────────────────────────────┐
│                    Browser Automation                       │
│  ┌─────────────┬──────────────┬─────────────────────────┐  │
│  │   Browser   │ DOMProcessor │        Tools            │  │
│  │   (trait)   │    (trait)   │                         │  │
│  └──────┬──────┴──────┬───────┴────────────────┬────────┘  │
│         │             │                      │           │
└─────────┼─────────────┼──────────────────────┼───────────┘
          │             │                      │
    ┌─────▼──────┐ ┌───▼────┐        ┌────────▼─────┐
    │  Browser   │ │DomSvc  │        │   Handlers   │
    │            │ │        │        │              │
    │TabManager  │ │CDP     │        │Navigation    │
    │NavManager  │ │HTML    │        │Interaction   │
    │Screenshot  │ │Tree    │        │Tabs          │
    │            │ │Builder │        │Content       │
    └────────────┘ └────────┘        └──────────────┘

Key Components

Component	Responsibility	Trait-Based
Browser	Manages browser session and lifecycle	Implements BrowserClient
DOMProcessor	Extracts and serializes DOM	Implements DOMProcessor
Tools	Action registry and execution	Uses BrowserClient trait
Handlers	Specific action implementations	Use ActionHandler trait

📁 Project Structure

browsing/
├── src/
│   ├── agent/              # Agent view types (used by tools layer)
│   │   ├── views.rs        # Data types
│   │   └── memory.rs       # Session memory types
│   ├── browser/            # Browser management
│   │   ├── session.rs      # Browser session (BrowserClient impl)
│   │   ├── tab_manager.rs  # Tab operations
│   │   ├── navigation.rs   # Navigation operations
│   │   ├── screenshot.rs   # Screenshot operations
│   │   ├── cdp.rs          # CDP WebSocket client
│   │   ├── launcher.rs     # Browser launcher
│   │   └── profile.rs      # Browser configuration
│   ├── dom/                # DOM processing
│   │   ├── processor.rs    # DOMProcessor trait impl
│   │   ├── serializer.rs   # Text serialization
│   │   ├── tree_builder.rs # DOM tree construction
│   │   ├── cdp_client.rs   # CDP wrapper for DOM
│   │   └── html_converter.rs # HTML to markdown
│   ├── tools/              # Action system
│   │   ├── service.rs      # Tools registry
│   │   ├── handlers/       # Action handlers
│   │   │   ├── navigation.rs
│   │   │   ├── interaction.rs
│   │   │   ├── tabs.rs
│   │   │   ├── content.rs
│   │   │   └── advanced.rs
│   │   └── params.rs       # Parameter extraction
│   ├── traits/             # Core trait abstractions
│   │   ├── browser_client.rs  # BrowserClient trait
│   │   └── dom_processor.rs   # DOMProcessor trait
│   ├── actor/              # Low-level interactions
│   │   ├── page.rs         # Page operations
│   │   ├── element.rs      # Element operations
│   │   ├── mouse.rs        # Mouse interactions
│   │   └── keyboard.rs     # Keyboard input
│   ├── config.rs           # Configuration
│   ├── error.rs            # Error types
│   ├── logging.rs          # Logging setup
│   ├── metrics.rs          # Metrics collection
│   ├── utils.rs            # Utilities
│   └── views.rs            # Shared data types
└── Cargo.toml

🎨 Design Principles

Trait-Facing Design

• BrowserClient - Abstract browser operations for testing and alternative backends
• DOMProcessor - Pluggable DOM processing implementations
• ActionHandler - Extensible action system

Separation of Concerns

• TabManager - Tab operations (create, switch, close)
• NavigationManager - Navigation logic
• ScreenshotManager - Screenshot capture
• Handlers - Focused action implementations

DRY (Don't Repeat Yourself)

• ActionParams - Reusable parameter extraction
• JSONExtractor - Centralized JSON parsing
• SessionGuard - Unified session access

KISS (Keep It Simple, Stupid)

• Split complex methods into focused helpers
• Clear naming and single responsibility
• Minimal dependencies between modules

🧪 Testing

# Run all tests
cargo test

# Run with output
cargo test -- --nocapture

# Run specific test
cargo test test_browser_navigation

# Run integration tests only
cargo test --test integration

Test Coverage

• 350+ tests across all modules (all passing, 0 ignored)
• Test files:
  • actor_test.rs - Page, Element, Mouse, Keyboard operations
  • browser_lifecycle_test.rs - Browser start/stop lifecycle
  • browser_managers_test.rs - Navigation, Screenshot, Tab managers
  • browser_test.rs - Browser integration
  • cdp_enhancements_test.rs - CDP client features
  • dom_extraction_test.rs - DOM extraction
  • dom_test.rs - DOM state and serialization
  • error_handling_test.rs - Error variants and propagation
  • error_recovery_test.rs - Config validation and recovery
  • integration_test.rs - Action models, URL extraction, tools
  • integration_workflow_test.rs - Full workflow
  • security_test.rs - Security validation
  • tools_handlers_test.rs - All action handlers
  • tools_test.rs - Tools registry
  • traits_test.rs - BrowserClient, DOMProcessor traits
  • utils_test.rs - URL extraction, signal handling
  • agent_test.rs - Agent view types
  • agent_service_test.rs - Agent state and history
• Mock implementations for deterministic testing
• Trait-based mocking for browser/DOM components

⚠️ Data Retention Policy

Browser Data is NEVER Deleted

IMPORTANT: The browsing library never deletes browser data for safety reasons.

What This Means:

Data Type	Behavior
Bookmarks	Never deleted
History	Never deleted
Cookies	Never deleted
Passwords	Never deleted
Extensions	Never deleted
Cache	Never deleted
Temp Directories	Never deleted (left in /tmp/)

Why This Policy Exists:

1. User Safety: Users may specify a custom user_data_dir pointing to their real browser profile
2. Catastrophe Prevention: Accidentally deleting a user's real browser data (bookmarks, history, passwords) would be devastating
3. Debugging: Leaving temp directories allows inspection after crashes or failures
4. User Control: Users are responsible for managing their own browser data

How It Works:

When no user_data_dir is specified:

let profile = BrowserProfile {
    user_data_dir: None,  // Uses temp directory: /tmp/browser-use-1738369200000/
    ..Default::default()
};

When browser.stop() is called:

• ✅ Browser process is killed
• ✅ In-memory state is cleared
• ❌ User data directory is NOT deleted

Managing Temporary Data:

Users are responsible for cleanup:

# List browser temp directories
ls -la /tmp/browser-use-*

# Delete old temp directories (optional, manual cleanup)
rm -rf /tmp/browser-use-1738369200000/

Using a Custom Data Directory:

let profile = BrowserProfile {
    user_data_dir: Some("/path/to/custom/profile".into()),
    ..Default::default()
};

Warning: If you point to your real browser profile, the library will NOT protect it. You're responsible for that directory.

🔧 Configuration

Browser Profile

use browsing::BrowserProfile;

// Default: headless mode with lightweight flags
let profile = BrowserProfile::default();

// Explicit headless
let profile = BrowserProfile::default().with_headless(true);

// Visible browser for debugging
let profile = BrowserProfile::default().with_headless(false);

// With custom data directory
let profile = BrowserProfile {
    user_data_dir: Some("/path/to/profile".into()),
    ..Default::default()
};

📖 API Documentation

Generate and view API docs:

cargo doc --open