web-capture 0.3.23

CLI and microservice to render web pages as HTML, Markdown, or PNG
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291

# web-capture (Rust)

[![crates.io version](https://img.shields.io/crates/v/web-capture?label=crates.io&color=orange)](https://crates.io/crates/web-capture)
[![crates.io downloads](https://img.shields.io/crates/d/web-capture?label=downloads&color=orange)](https://crates.io/crates/web-capture)
[![docs.rs](https://img.shields.io/docsrs/web-capture?label=docs.rs)](https://docs.rs/web-capture)
[![GitHub Release](https://img.shields.io/github/v/release/link-assistant/web-capture?label=GitHub%20Release)](https://github.com/link-assistant/web-capture/releases)
[![CI - Rust](https://img.shields.io/github/actions/workflow/status/link-assistant/web-capture/rust.yml?branch=main&label=CI)](https://github.com/link-assistant/web-capture/actions/workflows/rust.yml)
[![License: Unlicense](https://img.shields.io/badge/license-Unlicense-green)](https://unlicense.org)

A CLI and microservice to fetch URLs and render them as:

- **Markdown**: Converted from HTML with image extraction (default)
- **HTML**: Rendered page content
- **PNG screenshot**: Full page capture

This is the Rust implementation of web-capture, providing the same API as the JavaScript version.

## Installation

### From crates.io

```bash
cargo install web-capture
```

### From Source

```bash
cd rust
cargo build --release
```

## Quick Start

### CLI Usage

```bash
# Capture a URL as Markdown (default format)
# Output auto-derived to ./data/web-capture/<host>/<path>/document.md
web-capture https://example.com

# Capture as Markdown to a specific file
web-capture https://example.com -o page.md

# Write to stdout explicitly
web-capture https://example.com -o -

# Capture as HTML
web-capture https://example.com --format html

# Take a screenshot
web-capture https://example.com --format png -o screenshot.png

# Create a ZIP archive
web-capture https://example.com --archive

# Keep images inline (opt-in)
web-capture https://example.com --embed-images -o page.md

# Structured search-provider capture (JSON by default)
web-capture search "formal methods" --provider wikipedia

# Start as API server
web-capture --serve

# Start server on custom port
web-capture --serve --port 8080
```

### API Endpoints (Server Mode)

- **Markdown**: `GET /markdown?url=<URL>` (original links kept, base64 stripped by default)
- **Markdown (kreuzberg)**: `GET /markdown?url=<URL>&converter=kreuzberg`
- **Markdown (structured JSON)**: `GET /markdown?url=<URL>&converter=kreuzberg&format=json`
- **Markdown (base64 inline)**: `GET /markdown?url=<URL>&embedImages=true`
- **Markdown (all images stripped)**: `GET /markdown?url=<URL>&keepOriginalLinks=false`
- **HTML**: `GET /html?url=<URL>`
- **PNG screenshot**: `GET /image?url=<URL>`
- **Search**: `GET /search?q=<QUERY>&provider=<PROVIDER>&format=json|markdown`

### Search Endpoint

```
GET /search?q=<QUERY>&provider=<PROVIDER>&format=json|markdown
```

Captures structured results from a search provider in a normalized,
machine-readable shape. `wikipedia` (default) uses the CORS-friendly REST API;
the HTML engines (`duckduckgo`, `google`, `bing`, `brave`) are parsed
server-side. Blocked or CAPTCHA-gated pages are reported through `diagnostics`.

| Parameter  | Required | Description                                          | Default     |
| ---------- | -------- | ---------------------------------------------------- | ----------- |
| `q`        | Yes      | Search query (`query` accepted as an alias)          | -           |
| `provider` | No       | `wikipedia`, `duckduckgo`, `google`, `bing`, `brave` | `wikipedia` |
| `limit`    | No       | Maximum number of results                            | `10`        |
| `format`   | No       | Response format: `json` or `markdown`                | `json`      |

The JSON shape matches the JavaScript implementation:

```json
{
  "query": "formal methods",
  "provider": "wikipedia",
  "captureMode": "fetch",
  "capturedAt": "2026-05-18T20:30:00Z",
  "results": [
    {
      "rank": 1,
      "title": "Formal methods",
      "url": "https://en.wikipedia.org/wiki/Formal_methods",
      "snippet": "mathematically rigorous techniques for the specification..."
    }
  ],
  "diagnostics": {
    "status": 200,
    "blockedByCors": false,
    "blockedByCaptcha": false,
    "sourceUrl": "https://en.wikipedia.org/w/rest.php/v1/search/page?q=formal+methods&limit=10"
  }
}
```

## CLI Reference

### Server Mode

Start the API server:

```bash
web-capture --serve [--port <port>]
```

| Option    | Short | Description              | Default            |
| --------- | ----- | ------------------------ | ------------------ |
| `--serve` | `-s`  | Start as HTTP API server | -                  |
| `--port`  | `-p`  | Port to listen on        | 3000 (or PORT env) |

### Capture Mode

Capture a URL directly:

```bash
web-capture <url> [options]
```

| Option                      | Short | Description                                           | Default               |
| --------------------------- | ----- | ----------------------------------------------------- | --------------------- |
| `--format`                  | `-f`  | Output format: `markdown`/`md`, `html`, `image`/`png` | `markdown`            |
| `--output`                  | `-o`  | Output file path. Use `-o -` for stdout               | auto-derived from URL |
| `--capture`                 |       | Capture method: `browser` or `api`                    | `browser`             |
| `--data-dir`                |       | Base directory for auto-derived output paths          | `./data/web-capture`  |
| `--embed-images`            |       | Keep images as inline base64 data URIs                | false                 |
| `--no-extract-images`       |       | Alias for `--embed-images`                            | false                 |
| `--keep-original-links`     |       | Keep original remote URLs, strip base64               | false                 |
| `--images-dir`              |       | Subdirectory name for extracted images                | `images`              |
| `--archive`                 |       | Create archive: `zip`, `7z`, `tar.gz`, `tar`          | -                     |
| `--extract-latex`           |       | Extract LaTeX formulas                                | true                  |
| `--no-extract-latex`        |       | Disable LaTeX extraction                              | -                     |
| `--extract-metadata`        |       | Extract article metadata                              | true                  |
| `--no-extract-metadata`     |       | Disable metadata extraction                           | -                     |
| `--post-process`            |       | Apply post-processing                                 | true                  |
| `--no-post-process`         |       | Disable post-processing                               | -                     |
| `--detect-code-language`    |       | Detect code block languages                           | true                  |
| `--no-detect-code-language` |       | Disable code language detection                       | -                     |

### Search Mode

Capture structured search-provider results. Output defaults to JSON; pass
`--format markdown` for a human-readable document.

```bash
web-capture search "<query>" [options]
```

| Option       | Short | Description                                          | Default     |
| ------------ | ----- | ---------------------------------------------------- | ----------- |
| `--provider` |       | `wikipedia`, `duckduckgo`, `google`, `bing`, `brave` | `wikipedia` |
| `--limit`    |       | Maximum number of results                            | `10`        |
| `--format`   | `-f`  | Output format: `json` or `markdown`                  | `json`      |

### Examples

```bash
# Capture Markdown (default)
web-capture https://example.com

# Capture to specific file
web-capture https://example.com -o page.md

# Write to stdout
web-capture https://example.com -o -

# HTML format
web-capture https://example.com -f html -o page.html

# Google Docs live editor model
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture browser

# Google Docs public export endpoint
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture api

# Google Docs REST API with OAuth token
web-capture https://docs.google.com/document/d/DOC_ID/edit --capture api --api-token YOUR_TOKEN

# Tune browser-model quiescence for large or slow documents
WEB_CAPTURE_GDOCS_STABILITY_MS=2500 WEB_CAPTURE_GDOCS_MAX_WAIT_MS=60000 \
  web-capture https://docs.google.com/document/d/DOC_ID/edit --capture browser

# Screenshot
web-capture https://example.com -f png -o screenshot.png

# Pipe to another command
web-capture https://example.com -o - | grep "title"

# Structured search (JSON by default)
web-capture search "formal methods"

# Search DuckDuckGo, limit to 5 results, render as Markdown
web-capture search "formal methods" --provider duckduckgo --limit 5 --format markdown
```

## Docker

```bash
# Build and run
docker build -t web-capture-rust .
docker run -p 3000:3000 web-capture-rust
```

## Configuration

### Environment Variables

| Variable                           | Description                          | Default              |
| ---------------------------------- | ------------------------------------ | -------------------- |
| `PORT`                             | Server port                          | `3000`               |
| `API_TOKEN`                        | API token for authenticated capture  | -                    |
| `WEB_CAPTURE_DATA_DIR`             | Base directory for output            | `./data/web-capture` |
| `WEB_CAPTURE_EMBED_IMAGES`         | `0`/`1` — keep images inline         | `0`                  |
| `WEB_CAPTURE_KEEP_ORIGINAL_LINKS`  | `0`/`1` — keep original remote URLs  | `0`                  |
| `WEB_CAPTURE_IMAGES_DIR`           | Subdirectory for extracted images    | `images`             |
| `WEB_CAPTURE_EXTRACT_LATEX`        | `0`/`1` — extract LaTeX              | `1`                  |
| `WEB_CAPTURE_EXTRACT_METADATA`     | `0`/`1` — extract metadata           | `1`                  |
| `WEB_CAPTURE_POST_PROCESS`         | `0`/`1` — post-processing            | `1`                  |
| `WEB_CAPTURE_DETECT_CODE_LANGUAGE` | `0`/`1` — detect code langs          | `1`                  |
| `RUST_LOG`                         | Log level (e.g. `web_capture=debug`) | `web_capture=info`   |

## Library Usage

Add to your `Cargo.toml`:

```toml
[dependencies]
web-capture = "0.2"
```

### Example

```rust
use web_capture::{fetch_html, convert_html_to_markdown, capture_screenshot};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Fetch HTML from a URL
    let html = fetch_html("https://example.com").await?;
    println!("HTML length: {}", html.len());

    // Convert HTML to Markdown
    let markdown = convert_html_to_markdown(&html, Some("https://example.com"))?;
    println!("Markdown: {}", markdown);

    // Capture a screenshot
    let screenshot = capture_screenshot("https://example.com").await?;
    println!("Screenshot size: {} bytes", screenshot.len());

    Ok(())
}
```

## Built With

- [Axum](https://github.com/tokio-rs/axum) - Web framework
- [browser-commander](https://github.com/link-foundation/browser-commander) - Browser automation
- [html2md](https://github.com/nickyc975/html2md-rs) - HTML to Markdown conversion
- [scraper](https://github.com/causal-agent/scraper) - HTML parsing
- [Tokio](https://tokio.rs/) - Async runtime

## License

[Unlicense](../LICENSE) — This is free and unencumbered software released into the public domain. You are free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means. See [https://unlicense.org](https://unlicense.org) for details.