crw-cli 0.18.2

crw — Unified CLI for web scraping, crawling, search, and serving
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160

# crw-cli

Standalone CLI tool for scraping URLs to markdown, JSON, HTML, or plain text — no server needed.

[![crates.io](https://img.shields.io/crates/v/crw-cli.svg)](https://crates.io/crates/crw-cli)
[![license](https://img.shields.io/badge/license-AGPL--3.0-blue.svg)](https://github.com/us/crw/blob/main/LICENSE)

## Overview

`crw-cli` is a single-binary web scraper that fetches any URL and outputs clean content to stdout. Part of the [CRW](https://github.com/us/crw) project — same extraction engine as the server, but with zero setup.

- **6 output formats** — markdown, JSON, HTML, raw HTML, plain text, links
- **Main content extraction** — automatically strips nav, footer, ads, scripts
- **CSS selector & XPath** — extract specific elements before conversion
- **Stealth mode** — User-Agent rotation and browser-like headers
- **JS rendering** — optional CDP-based rendering for SPAs (via `--js` flag)
- **Proxy support** — per-request HTTP, HTTPS, or SOCKS5 proxy
- **File output** — write directly to a file with `-o`

## Installation

```bash
cargo install crw-cli
```

This installs the `crw` binary.

## Usage

### Basic scraping

```bash
# Scrape a page to markdown (default)
crw https://example.com

# Output as JSON (includes all metadata)
crw https://example.com --format json

# Output as plain text
crw https://example.com --format text

# Output as HTML (cleaned)
crw https://example.com --format html

# Output raw HTML (no cleaning)
crw https://example.com --format rawhtml

# Extract all links
crw https://example.com --format links
```

### CSS selector extraction

Extract only specific elements:

```bash
# Extract just the article content
crw https://blog.example.com --css 'article.post'

# Extract the main heading
crw https://example.com --css 'h1'
```

### XPath extraction

```bash
# Extract all paragraph text
crw https://example.com --xpath '//p'

# Extract a specific element by ID
crw https://example.com --xpath '//*[@id="content"]'
```

### Save to file

```bash
crw https://example.com -o page.md
crw https://example.com --format json -o page.json
```

### Full page content (no main content extraction)

By default, `crw` strips boilerplate (nav, footer, ads). Use `--raw` to get everything:

```bash
crw https://example.com --raw
```

### Stealth mode

Rotate User-Agent and inject browser-like headers to reduce bot detection:

```bash
crw https://protected-site.com --stealth
```

### Proxy

Route requests through a proxy:

```bash
crw https://example.com --proxy http://user:pass@proxy:8080
```

### JavaScript rendering

For SPAs that require JavaScript, use `--js`. The CLI auto-detects a locally
installed LightPanda or Chrome; you can also point at an existing CDP WebSocket
with `CRW_CDP_URL`:

```bash
# Auto-detect (spawns LightPanda/Chrome if installed)
crw https://spa-app.com --js

# Or point at a running CDP endpoint
CRW_CDP_URL=ws://127.0.0.1:9222 crw https://spa-app.com --js
```

Note: `CRW_CDP_URL` is only honored by `crw` (the CLI). In server/MCP mode use
`[renderer.lightpanda.ws_url]` / `[renderer.chrome.ws_url]` (or the matching
`CRW_RENDERER__*__WS_URL` env vars).

## All options

```
Usage: crw [OPTIONS] <URL>

Arguments:
  <URL>  URL to scrape (http or https)

Options:
  -f, --format <FORMAT>      Output format [default: markdown]
                              [values: markdown, json, html, rawhtml, text, links]
  -o, --output <FILE>        Write output to file instead of stdout
      --raw                  Disable main content extraction (full page)
      --js                   Force JS rendering (auto-detects LightPanda/Chrome, or CRW_CDP_URL)
      --css <SELECTOR>       Extract only elements matching this CSS selector
      --xpath <EXPR>         Extract only elements matching this XPath expression
      --proxy <URL>          HTTP, HTTPS, or SOCKS5 proxy URL (e.g. socks5://user:pass@host:1080)
      --stealth              Enable stealth mode (UA rotation + browser headers)
  -h, --help                 Print help
```

## Part of CRW

This crate is part of the [CRW](https://github.com/us/crw) workspace — a fast, lightweight, Firecrawl-compatible web scraper built in Rust.

| Crate | Description |
|-------|-------------|
| [crw-core](https://crates.io/crates/crw-core) | Core types, config, and error handling |
| [crw-renderer](https://crates.io/crates/crw-renderer) | HTTP + CDP browser rendering engine |
| [crw-extract](https://crates.io/crates/crw-extract) | HTML → markdown/plaintext extraction |
| [crw-crawl](https://crates.io/crates/crw-crawl) | Async BFS crawler with robots.txt & sitemap |
| [crw-server](https://crates.io/crates/crw-server) | Firecrawl-compatible API server |
| **crw-cli** | Standalone CLI — `crw` binary (this crate) |
| [crw-mcp](https://crates.io/crates/crw-mcp) | MCP stdio proxy binary |

## License

AGPL-3.0 — see [LICENSE](https://github.com/us/crw/blob/main/LICENSE).