fetchkit 0.6.0

AI-friendly web content fetching and HTML-to-Markdown conversion library
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
# fetchkit

AI-friendly web content fetching tool designed for LLM consumption. Rust library with CLI, MCP server, and Python bindings.

## Features

- **HTTP fetching** - GET and HEAD methods with streaming support
- **Pluggable fetchers** - URL-aware dispatch to specialized handlers for repos, docs, feeds, videos, papers, and more
- **HTML-to-Markdown** - Built-in conversion optimized for LLMs, with fetched relative links/images resolved to absolute URLs
- **Agent content focus** - Optional low-noise extraction mode for AI agents
- **Crawl discovery** - Optional bounded same-origin page discovery for AI agents
- **HTML-to-Text** - Plain text extraction with clean formatting
- **Content processors** - Post-download extraction for text PDFs and Office documents, with an extensible registry
- **Binary detection** - Returns metadata only for unsupported binary formats
- **Timeout handling** - 1s first-byte, 30s body with partial content on timeout
- **Safety limits** - 10 MB default decompressed body cap with truncation
- **URL filtering** - URL-aware allow/block lists for controlled access
- **SSRF protection** - Resolve-then-check blocks private IPs by default
- **MCP server** - Model Context Protocol support for AI tool integration

## Built-in Fetchers

Fetchkit routes each request through an ordered fetcher registry. Specialized
fetchers match first; the default fetcher handles everything else.

- `GitHubCodeFetcher` - GitHub source file URLs (`/blob/...`)
- `GitHubIssueFetcher` - GitHub issue and pull request URLs
- `GitHubRepoFetcher` - GitHub repository home pages
- `TwitterFetcher` - X/Twitter status URLs
- `StackOverflowFetcher` - Stack Overflow and Stack Exchange question URLs
- `PackageRegistryFetcher` - PyPI, crates.io, and npm package pages
- `WikipediaFetcher` - Wikipedia article URLs
- `YouTubeFetcher` - YouTube watch and `youtu.be` URLs
- `ArXivFetcher` - arXiv abstract and PDF URLs
- `HackerNewsFetcher` - Hacker News item threads
- `RSSFeedFetcher` - RSS and Atom feed URLs
- `DocsSiteFetcher` - docs sites with `llms.txt`/`llms-full.txt` support
- `DefaultFetcher` - all remaining HTTP/HTTPS URLs with HTML conversion, streaming, timeout handling, and binary detection

## Built-in Content Processors

Content processors run after a fetcher retrieves a bounded response body. They
select by final URL, response media type, and requested output, then turn
documents into LLM-friendly content without performing their own network
requests.

- `HtmlProcessor` - metadata and focused-content extraction followed by native
  Markdown or text conversion; accepts a custom `HtmlToMarkdownConverter`
- `DocumentProcessor` - PDF and Office documents to Markdown through a
  configurable chain of document converters (see [Document conversion]#document-conversion)

Custom processors implement `ContentProcessor` and are added with
`ToolBuilder::content_processor` (checked before the built-ins) or registered in a
`ContentProcessorRegistry` passed to `FetcherRegistry::with_content_processors`.

## Document Conversion

Documents are always downloaded by fetchkit itself, under the same URL policy,
DNS pinning, transport and timeouts as any other fetch. `DocumentProcessor` then
detects the format (Content-Type, then file extension for generic binary types,
confirmed by magic bytes) and runs an ordered chain of `DocumentConverter`
backends. The next converter runs when one fails, does not support the input, or
returns nothing usable (for example a scanned PDF that needs OCR).

| Converter | Cargo feature | Default | Where it runs | Formats |
| --- | --- | --- | --- | --- |
| `PdfInspectorConverter` (`pdf-inspector`) | `docs-pdf` | on | local | PDF (text layer; reports OCR-only pages) |
| `MarkkitConverter` (`markkit`) | `docs-markkit` | on | local | DOCX, XLSX, XLSM, XLS, ODS (via [markkit]https://crates.io/crates/markkit) |
| `OfficeConverter` (`office`), deprecated | `docs-office` | off | local | DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS |
| `CloudflareConverter` (`cloudflare`) | `docs-cloudflare` | off | remote | PDF, DOCX, XLSX, XLSB, XLS, ODT, ODS |
| `HttpConverter` (any name) | `docs-http` | off | remote (configurable) | any format your service accepts |

`markkit` is the default Office backend. The in-crate `office` converter is
deprecated and kept for one release behind the non-default `docs-office` feature;
it is still the only built-in PPTX and XLSB converter. With both features enabled,
`markkit` runs first and `office` is the fallback. `docs-all` enables every
backend except the deprecated `docs-office`. Tune markkit limits (rows and
columns per sheet, hidden sheets, archive budgets) by registering
`MarkkitConverter::with_markkit(Markkit::builder()...build())`, which replaces the
default `markkit` converter in place. Remote converters upload document bytes to a
third party, so they only run when remote conversion is explicitly allowed, and
never under `ToolBuilder::hardened()`.

```rust
use fetchkit::documents::CloudflareConverter;
use fetchkit::{DocumentConversion, DocumentFormat, ToolBuilder};

let tool = ToolBuilder::new()
    .document_conversion(
        DocumentConversion::default()                 // pdf-inspector + markkit
            .converter(CloudflareConverter::new(account_id, api_token))
            .allow_remote(true)                       // opt in to uploads
            .chain(DocumentFormat::Pdf, ["pdf-inspector", "cloudflare"])
            .formats([DocumentFormat::Pdf, DocumentFormat::Docx, DocumentFormat::Xlsx])
            .max_document_bytes(25 * 1024 * 1024),    // separate from max_body_size
    )
    .build();
```

Extension points: implement `DocumentConverter` for a new backend,
`FormatDetector` to change format detection, `ConverterClient` to route remote
uploads through your own egress, or `ContentProcessor` to bypass the chain.

CLI and MCP server flags: `--convert pdf,docx`, `--converter markkit,cloudflare`,
`--allow-remote-conversion`, `--max-document-bytes`. The Cloudflare converter
reads `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` (build with
`--features docs-cloudflare`). Python: `FetchkitTool(document_formats=...,
converters=..., allow_remote_conversion=..., max_document_bytes=...,
cloudflare_account_id=..., cloudflare_api_token=...)`.

## Installation

### From crates.io (recommended)

```bash
cargo install fetchkit-cli
```

### From Git

```bash
cargo install --git https://github.com/everruns/fetchkit fetchkit-cli
```

### From Source

```bash
git clone https://github.com/everruns/fetchkit
cd fetchkit
cargo install --path crates/fetchkit-cli
```

## CLI Usage

```bash
# Fetch URL (outputs markdown with frontmatter)
fetchkit fetch https://example.com

# Output as JSON instead
fetchkit fetch https://example.com -o json

# Custom user agent
fetchkit fetch https://example.com --user-agent "MyBot/1.0"

# Hardened outbound policy for cluster/data-plane use
fetchkit fetch https://example.com --hardened

# Discover a small same-origin page map for an agent
fetchkit fetch https://example.com --content-focus agent --crawl --max-pages 5

# Optional JS/DOM rendering for simple SPAs/docs (requires render-rakers feature)
fetchkit fetch https://example.com/app --render-rakers

# Show full documentation
fetchkit --llmtxt
```

Default output is markdown with YAML frontmatter:

```markdown
---
url: https://example.com
status_code: 200
source_content_type: text/html; charset=UTF-8
source_size: 1256
quality_score: 1.00
extraction_method: "full"
---
# Example Domain

This domain is for use in illustrative examples in documents...
```

JSON output (`-o json`):

```json
{
  "url": "https://example.com",
  "status_code": 200,
  "content_type": "text/html",
  "size": 1256,
  "format": "markdown",
  "content": "# Example Domain\n\nThis domain is for use in illustrative examples..."
}
```

## MCP Server

Run as a Model Context Protocol server:

```bash
fetchkit mcp

# Hardened profile for cluster/data-plane use
fetchkit mcp --hardened
```

Exposes `fetchkit` tool over JSON-RPC 2.0 stdio transport. Returns markdown with frontmatter (same format as CLI). Compatible with Claude Desktop and other MCP clients.

## Library Usage

Add to `Cargo.toml`:

```toml
[dependencies]
fetchkit = "0.2"
```

Optional rendered fetching:

```toml
[dependencies]
fetchkit = { version = "0.2", features = ["render-rakers"] }
```

`render-rakers` is not enabled by default. It is lightweight partial rendering:
inline JavaScript can update the DOM before markdown/text conversion, but it is
not a full browser engine. FetchKit blocks rakers-initiated subresource network
access in this mode; the initial page still uses FetchKit's normal URL, DNS,
proxy, timeout, and size policies.

### Basic Fetch

```rust
use fetchkit::{fetch, FetchRequest};

#[tokio::main]
async fn main() {
    let request = FetchRequest::new("https://example.com").as_markdown();

    let response = fetch(request).await.unwrap();
    println!("{}", response.content.unwrap_or_default());
}
```

### With Tool Builder

```rust
use fetchkit::{FetchRequest, ToolBuilder};

let tool = ToolBuilder::new()
    .enable_markdown(true)
    .enable_text(false)
    .user_agent("MyBot/1.0")
    .allow_prefix("https://docs.example.com")
    .block_prefix("https://internal.example.com")
    .build();

let request = FetchRequest::new("https://example.com");
let response = tool.execute(request).await.unwrap();
```

### Toolkit Contract Surface

```rust
use fetchkit::ToolBuilder;

let builder = ToolBuilder::new().enable_save_to_file(true);
let tool = builder.build();

assert_eq!(tool.name(), "web_fetch");
assert_eq!(tool.display_name(), "Web Fetch");

let definition = builder.build_tool_definition();
let mut service = builder.build_service();
```

### Hardened Tool Profile

```rust
use fetchkit::Tool;

let tool = Tool::builder()
    .hardened()
    .allow_prefix("https://docs.example.com")
    .build();
```

## Python Bindings

```bash
pip install fetchkit
```

```python
from fetchkit_py import fetch, FetchRequest, FetchkitTool

# Simple fetch
response = fetch("https://example.com", as_markdown=True)
print(response.content)

# With configuration
tool = FetchkitTool(
    enable_markdown=True,
    user_agent="MyBot/1.0",
    allow_prefixes=["https://docs.example.com"]
)
response = tool.fetch("https://example.com")
```

## Request Fields

| Field | Type | Description |
|-------|------|-------------|
| `url` | string | URL to fetch (required, `http://` or `https://`) |
| `method` | enum? | `GET` (default) or `HEAD` |
| `as_markdown` | bool? | Convert HTML to markdown |
| `as_text` | bool? | Convert HTML to plain text |
| `save_to_file` | string? | Non-blank destination path; validated by `FileSaver` before fetching |
| `content_focus` | string? | `"full"`/unset returns everything; `"main"` strips semantic boilerplate; `"readable"` selects article-like content; `"agent"` selects the best low-noise strategy for AI agents |
| `crawl` | bool? | Fetch the seed URL, then discover and fetch bounded same-origin pages |
| `max_pages` | int? | Maximum crawl pages, including the seed; default 5, max 20 |
| `if_none_match` | string? | ETag for conditional `If-None-Match` |
| `if_modified_since` | string? | Timestamp for conditional `If-Modified-Since` |
| `render` | string? | `"rakers"` to opt into rendered fetch when enabled |

## Response Fields

| Field | Type | Description |
|-------|------|-------------|
| `url` | string | Fetched URL |
| `status_code` | int | HTTP status code |
| `content_type` | string? | Content-Type header |
| `size` | int? | Content size in bytes |
| `last_modified` | string? | Last-Modified header |
| `etag` | string? | ETag header (use for next conditional request) |
| `filename` | string? | From Content-Disposition |
| `format` | string? | `"markdown"`, `"text"`, `"raw"`, or a fetcher-specific format |
| `content` | string? | Page content |
| `truncated` | bool? | True if content was cut off |
| `method` | string? | `"HEAD"` for HEAD requests |
| `error` | string? | Error message if failed |
| `saved_path` | string? | Filesystem path when `save_to_file` succeeded |
| `bytes_written` | int? | Bytes saved to file |
| `metadata` | object? | Structured `PageMetadata` (title, description, links, headings, extraction method, …) |
| `quality` | object? | Agent-facing `PageQuality` (score, warnings, link density, suggested next action) |
| `crawl` | object? | Bounded crawl discovery result with visited page summaries |
| `word_count` | int? | Word count of returned content |
| `redirect_chain` | string[] | URLs visited during redirects (empty if none) |
| `is_paywall` | bool? | Heuristic paywall signal (soft, not guaranteed) |
| `rendered_by` | string? | Rendering backend used before conversion, e.g. `"rakers"` |

## Error Handling

Errors are returned in the `error` field:

- `InvalidUrl` - Malformed URL
- `UrlBlocked` - URL blocked by filter
- `NetworkError` - Connection failed
- `Timeout` - Request timed out
- `HttpError` - 4xx/5xx response
- `ContentError` - Failed to read body
- `BinaryContent` - Binary content has no registered processor

## Security

Fetchkit blocks connections to private/reserved IP ranges by default, preventing SSRF attacks when used in server-side or AI agent contexts.

**Blocked by default:** loopback, private networks (10.x, 172.16-31.x, 192.168.x), link-local (169.254.x including cloud metadata), IPv6 equivalents, multicast, and other reserved ranges.

```rust
// Default: private IPs blocked (safe for production)
let tool = Tool::default();

// Explicit opt-out for local development only
let tool = Tool::builder()
    .block_private_ips(false)
    .build();
```

DNS pinning prevents DNS rebinding attacks. IPv6-mapped IPv4 addresses are canonicalized before validation.
Redirects are followed manually in the default fetcher so each hop is revalidated against scheme and DNS policy. Allow/block prefixes are matched against parsed URLs rather than raw strings, which prevents lookalike host overmatches such as `allowed.example.com.evil.test`.
Proxy environment variables are ignored by default. Use the hardened profile for cluster-facing deployments and opt in with `ToolBuilder::respect_proxy_env(true)` only when it is part of an intentional egress design.

See the [`knowledge/security/threat-model.md`](knowledge/security/threat-model.md) concept for the full threat model.
See [`docs/hardening.md`](docs/hardening.md) for deployment guidance.

## Configuration

### Timeouts And Limits

- **First-byte**: 1 second (connect + initial response)
- **Body**: 30 seconds total
- **Body size**: 10 MB decompressed content by default

Partial content is returned on body timeout or body-size limit with `truncated: true`.

### PDF And Binary Content

When Markdown is requested, text-based PDFs are downloaded within the document
size limit (25 MB default) and converted to Markdown. PDF parsing runs locally; no OCR service or
additional network request is used. Scanned/image-only PDFs report `use_ocr` as the
suggested next action.

Office documents are converted the same way, locally: Word headings, lists and
tables, one section per PowerPoint slide, and one Markdown table per spreadsheet
sheet (DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS). Legacy DOC and PPT files are not
converted by the built-in local converters; route them to a remote or custom
converter if needed.

Other binary content returns metadata only:
- Images, audio, video, fonts
- Archives (zip, tar, rar, 7z)
- Legacy Word and PowerPoint files (DOC, PPT)

### HTML Conversion

HTML is automatically converted to markdown:
- Headers: `h1-h6` → `#` to `######`
- Lists: Proper nesting with 2-space indent
- Code: Language-aware, collision-safe fences and inline backticks
- Links/images: Titles, relative-URL resolution, and highest-resolution `srcset`
- Tables: Valid Markdown with formatted cells and escaped pipes
- Rich content: Footnotes, LaTeX math, callouts, figures, highlights,
  strikethrough, details, and task checkboxes
- Strips: document head, scripts, styles, templates, iframes, SVGs
- Adds a bounded [Agent resources]docs/agent-discoverability.md navigation appendix
  when discoverable resources are available

## License

MIT. See [Third-Party Notices](THIRD_PARTY_NOTICES.md) for adapted components.