web-capture (Rust)

A CLI and microservice to fetch URLs and render them as:

• Markdown: Converted from HTML with image extraction (default)
• HTML: Rendered page content
• PNG screenshot: Full page capture

This is the Rust implementation of web-capture, providing the same API as the JavaScript version.

Installation

From crates.io

cargo install web-capture

From Source

cd rust
cargo build --release

Quick Start

CLI Usage

# Capture a URL as Markdown (default format)
# Output auto-derived to ./data/web-capture/<host>/<path>/document.md
web-capture https://example.com

# Capture as Markdown to a specific file
web-capture https://example.com -o page.md

# Write to stdout explicitly
web-capture https://example.com -o -

# Capture as HTML
web-capture https://example.com --format html

# Take a screenshot
web-capture https://example.com --format png -o screenshot.png

# Create a ZIP archive
web-capture https://example.com --archive

# Keep images inline (opt-in)
web-capture https://example.com --embed-images -o page.md

# Start as API server
web-capture --serve

# Start server on custom port
web-capture --serve --port 8080

API Endpoints (Server Mode)

• Markdown: GET /markdown?url=<URL> (original links kept, base64 stripped by default)
• Markdown (base64 inline): GET /markdown?url=<URL>&embedImages=true
• Markdown (all images stripped): GET /markdown?url=<URL>&keepOriginalLinks=false
• HTML: GET /html?url=<URL>
• PNG screenshot: GET /image?url=<URL>

CLI Reference

Server Mode

Start the API server:

web-capture --serve [--port <port>]

Capture Mode

Capture a URL directly:

web-capture <url> [options]

Examples

# Capture Markdown (default)
web-capture https://example.com

# Capture to specific file
web-capture https://example.com -o page.md

# Write to stdout
web-capture https://example.com -o -

# HTML format
web-capture https://example.com -f html -o page.html

# Google Docs live editor model
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture browser

# Google Docs public export endpoint
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture api

# Google Docs REST API with OAuth token
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture api --api-token YOUR_TOKEN

# Screenshot
web-capture https://example.com -f png -o screenshot.png

# Pipe to another command
web-capture https://example.com -o - | grep "title"

Docker

# Build and run
docker build -t web-capture-rust .
docker run -p 3000:3000 web-capture-rust

Configuration

Environment Variables

Library Usage

Add to your Cargo.toml:

[dependencies]
web-capture = "0.2"

Example

use web_capture::{fetch_html, convert_html_to_markdown, capture_screenshot};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Fetch HTML from a URL
    let html = fetch_html("https://example.com").await?;
    println!("HTML length: {}", html.len());

    // Convert HTML to Markdown
    let markdown = convert_html_to_markdown(&html, Some("https://example.com"))?;
    println!("Markdown: {}", markdown);

    // Capture a screenshot
    let screenshot = capture_screenshot("https://example.com").await?;
    println!("Screenshot size: {} bytes", screenshot.len());

    Ok(())
}

Built With

• Axum - Web framework
• browser-commander - Browser automation
• html2md - HTML to Markdown conversion
• scraper - HTML parsing
• Tokio - Async runtime

License

Unlicense — This is free and unencumbered software released into the public domain. You are free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means. See https://unlicense.org for details.