web-capture (Rust)

A CLI and microservice to fetch URLs and render them as:

• Markdown: Converted from HTML with image extraction (default)
• HTML: Rendered page content
• PNG screenshot: Full page capture

This is the Rust implementation of web-capture, providing the same API as the JavaScript version.

Installation

From crates.io

cargo install web-capture

From Source

cd rust
cargo build --release

Quick Start

CLI Usage

# Capture a URL as Markdown (default format)
# Output auto-derived to ./data/web-capture/<host>/<path>/document.md
web-capture https://example.com

# Capture as Markdown to a specific file
web-capture https://example.com -o page.md

# Write to stdout explicitly
web-capture https://example.com -o -

# Capture as HTML
web-capture https://example.com --format html

# Take a screenshot
web-capture https://example.com --format png -o screenshot.png

# Create a ZIP archive
web-capture https://example.com --archive

# Keep images inline (opt-in)
web-capture https://example.com --embed-images -o page.md

# Structured search-provider capture (JSON by default)
web-capture search "formal methods" --provider wikipedia

# Start as API server
web-capture --serve

# Start server on custom port
web-capture --serve --port 8080

API Endpoints (Server Mode)

• Markdown: GET /markdown?url=<URL> (original links kept, base64 stripped by default)
• Markdown (base64 inline): GET /markdown?url=<URL>&embedImages=true
• Markdown (all images stripped): GET /markdown?url=<URL>&keepOriginalLinks=false
• HTML: GET /html?url=<URL>
• PNG screenshot: GET /image?url=<URL>
• Search: GET /search?q=<QUERY>&provider=<PROVIDER>&format=json|markdown

Search Endpoint

GET /search?q=<QUERY>&provider=<PROVIDER>&format=json|markdown

Captures structured results from a search provider in a normalized, machine-readable shape. wikipedia (default) uses the CORS-friendly REST API; the HTML engines (duckduckgo, google, bing, brave) are parsed server-side. Blocked or CAPTCHA-gated pages are reported through diagnostics.

The JSON shape matches the JavaScript implementation:

{
  "query": "formal methods",
  "provider": "wikipedia",
  "captureMode": "fetch",
  "capturedAt": "2026-05-18T20:30:00Z",
  "results": [
    {
      "rank": 1,
      "title": "Formal methods",
      "url": "https://en.wikipedia.org/wiki/Formal_methods",
      "snippet": "mathematically rigorous techniques for the specification..."
    }
  ],
  "diagnostics": {
    "status": 200,
    "blockedByCors": false,
    "blockedByCaptcha": false,
    "sourceUrl": "https://en.wikipedia.org/w/rest.php/v1/search/page?q=formal+methods&limit=10"
  }
}

CLI Reference

Server Mode

Start the API server:

web-capture --serve [--port <port>]

Capture Mode

Capture a URL directly:

web-capture <url> [options]

Search Mode

Capture structured search-provider results. Output defaults to JSON; pass --format markdown for a human-readable document.

web-capture search "<query>" [options]

Examples

# Capture Markdown (default)
web-capture https://example.com

# Capture to specific file
web-capture https://example.com -o page.md

# Write to stdout
web-capture https://example.com -o -

# HTML format
web-capture https://example.com -f html -o page.html

# Google Docs live editor model
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture browser

# Google Docs public export endpoint
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture api

# Google Docs REST API with OAuth token
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture api --api-token YOUR_TOKEN

# Tune browser-model quiescence for large or slow documents
WEB_CAPTURE_GDOCS_STABILITY_MS=2500 WEB_CAPTURE_GDOCS_MAX_WAIT_MS=60000 \
  web-capture https://docs.google.com/document/d/DOC_ID/edit --capture browser

# Screenshot
web-capture https://example.com -f png -o screenshot.png

# Pipe to another command
web-capture https://example.com -o - | grep "title"

# Structured search (JSON by default)
web-capture search "formal methods"

# Search DuckDuckGo, limit to 5 results, render as Markdown
web-capture search "formal methods" --provider duckduckgo --limit 5 --format markdown

Docker

# Build and run
docker build -t web-capture-rust .
docker run -p 3000:3000 web-capture-rust

Configuration

Environment Variables

Library Usage

Add to your Cargo.toml:

[dependencies]
web-capture = "0.2"

Example

use web_capture::{fetch_html, convert_html_to_markdown, capture_screenshot};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Fetch HTML from a URL
    let html = fetch_html("https://example.com").await?;
    println!("HTML length: {}", html.len());

    // Convert HTML to Markdown
    let markdown = convert_html_to_markdown(&html, Some("https://example.com"))?;
    println!("Markdown: {}", markdown);

    // Capture a screenshot
    let screenshot = capture_screenshot("https://example.com").await?;
    println!("Screenshot size: {} bytes", screenshot.len());

    Ok(())
}

Built With

• Axum - Web framework
• browser-commander - Browser automation
• html2md - HTML to Markdown conversion
• scraper - HTML parsing
• Tokio - Async runtime

License

Unlicense — This is free and unencumbered software released into the public domain. You are free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means. See https://unlicense.org for details.