# fetchkit
AI-friendly web content fetching tool designed for LLM consumption. Rust library with CLI, MCP server, and Python bindings.
## Features
- **HTTP fetching** - GET and HEAD methods with streaming support
- **Pluggable fetchers** - URL-aware dispatch to specialized handlers for repos, docs, feeds, videos, papers, and more
- **HTML-to-Markdown** - Built-in conversion optimized for LLMs, with fetched relative links/images resolved to absolute URLs
- **Agent content focus** - Optional low-noise extraction mode for AI agents
- **Crawl discovery** - Optional bounded same-origin page discovery for AI agents
- **HTML-to-Text** - Plain text extraction with clean formatting
- **Content processors** - Post-download extraction for text PDFs and Office documents, with an extensible registry
- **Binary detection** - Returns metadata only for unsupported binary formats
- **Timeout handling** - 1s first-byte, 30s body with partial content on timeout
- **Safety limits** - 10 MB default decompressed body cap with truncation
- **URL filtering** - URL-aware allow/block lists for controlled access
- **SSRF protection** - Resolve-then-check blocks private IPs by default
- **MCP server** - Model Context Protocol support for AI tool integration
## Built-in Fetchers
Fetchkit routes each request through an ordered fetcher registry. Specialized
fetchers match first; the default fetcher handles everything else.
- `GitHubCodeFetcher` - GitHub source file URLs (`/blob/...`)
- `GitHubIssueFetcher` - GitHub issue and pull request URLs
- `GitHubRepoFetcher` - GitHub repository home pages
- `TwitterFetcher` - X/Twitter status URLs
- `StackOverflowFetcher` - Stack Overflow and Stack Exchange question URLs
- `PackageRegistryFetcher` - PyPI, crates.io, and npm package pages
- `WikipediaFetcher` - Wikipedia article URLs
- `YouTubeFetcher` - YouTube watch, Shorts, live, embed, and `youtu.be` URLs
- `SocialMediaFetcher` - TikTok and Vimeo videos; SoundCloud tracks and playlists
- `ArXivFetcher` - arXiv abstract and PDF URLs
- `HackerNewsFetcher` - Hacker News item threads
- `RSSFeedFetcher` - RSS and Atom feed URLs
- `DocsSiteFetcher` - docs sites with `llms.txt`/`llms-full.txt` support
- `DefaultFetcher` - all remaining HTTP/HTTPS URLs with HTML conversion, streaming, timeout handling, and binary detection
See [Social media fetching](docs/social-media.md) for supported URLs and metadata limits.
## Built-in Content Processors
Content processors run after a fetcher retrieves a bounded response body. They
select by final URL, response media type, and requested output, then turn
documents into LLM-friendly content without performing their own network
requests.
- `HtmlProcessor` - metadata and focused-content extraction followed by native
Markdown or text conversion; accepts a custom `HtmlToMarkdownConverter`
- `DocumentProcessor` - PDF and Office documents to Markdown through a
configurable chain of document converters (see [Document conversion](#document-conversion))
Custom processors implement `ContentProcessor` and are added with
`ToolBuilder::content_processor` (checked before the built-ins) or registered in a
`ContentProcessorRegistry` passed to `FetcherRegistry::with_content_processors`.
## Document Conversion
Documents are always downloaded by fetchkit itself, under the same URL policy,
DNS pinning, transport and timeouts as any other fetch. `DocumentProcessor` then
detects the format (Content-Type, then file extension for generic binary types,
confirmed by magic bytes) and runs an ordered chain of `DocumentConverter`
backends. The next converter runs when one fails, does not support the input, or
returns nothing usable (for example a scanned PDF that needs OCR).
| `PdfInspectorConverter` (`pdf-inspector`) | `docs-pdf` | on | local | PDF (text layer; reports OCR-only pages) |
| `MarkkitConverter` (`markkit`) | `docs-markkit` | on | local | DOCX, XLSX, XLSM, XLS, ODS (via [markkit](https://crates.io/crates/markkit)) |
| `OfficeConverter` (`office`), deprecated | `docs-office` | off | local | DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS |
| `CloudflareConverter` (`cloudflare`) | `docs-cloudflare` | off | remote | PDF, DOCX, XLSX, XLSB, XLS, ODT, ODS |
| `HttpConverter` (any name) | `docs-http` | off | remote (configurable) | any format your service accepts |
`markkit` is the default Office backend. The in-crate `office` converter is
deprecated and kept for one release behind the non-default `docs-office` feature;
it is still the only built-in PPTX and XLSB converter. With both features enabled,
`markkit` runs first and `office` is the fallback. `docs-all` enables every
backend except the deprecated `docs-office`. Tune markkit limits (rows and
columns per sheet, hidden sheets, archive budgets) by registering
`MarkkitConverter::with_markkit(Markkit::builder()...build())`, which replaces the
default `markkit` converter in place. Remote converters upload document bytes to a
third party, so they only run when remote conversion is explicitly allowed, and
never under `ToolBuilder::hardened()`.
```rust
use fetchkit::documents::CloudflareConverter;
use fetchkit::{DocumentConversion, DocumentFormat, ToolBuilder};
let tool = ToolBuilder::new()
.document_conversion(
DocumentConversion::default() // pdf-inspector + markkit
.converter(CloudflareConverter::new(account_id, api_token))
.allow_remote(true) // opt in to uploads
.chain(DocumentFormat::Pdf, ["pdf-inspector", "cloudflare"])
.formats([DocumentFormat::Pdf, DocumentFormat::Docx, DocumentFormat::Xlsx])
.max_document_bytes(25 * 1024 * 1024), // separate from max_body_size
)
.build();
```
Extension points: implement `DocumentConverter` for a new backend,
`FormatDetector` to change format detection, `ConverterClient` to route remote
uploads through your own egress, or `ContentProcessor` to bypass the chain.
CLI and MCP server flags: `--convert pdf,docx`, `--converter markkit,cloudflare`,
`--allow-remote-conversion`, `--max-document-bytes`. The Cloudflare converter
reads `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` (build with
`--features docs-cloudflare`). Python: `FetchkitTool(document_formats=...,
converters=..., allow_remote_conversion=..., max_document_bytes=...,
cloudflare_account_id=..., cloudflare_api_token=...)`.
## Installation
### From crates.io (recommended)
```bash
cargo install fetchkit-cli
```
### From Git
```bash
cargo install --git https://github.com/everruns/fetchkit fetchkit-cli
```
### From Source
```bash
git clone https://github.com/everruns/fetchkit
cd fetchkit
cargo install --path crates/fetchkit-cli
```
## CLI Usage
```bash
# Fetch URL (outputs markdown with frontmatter)
fetchkit fetch https://example.com
# Output as JSON instead
fetchkit fetch https://example.com -o json
# Custom user agent
fetchkit fetch https://example.com --user-agent "MyBot/1.0"
# Hardened outbound policy for cluster/data-plane use
fetchkit fetch https://example.com --hardened
# Discover a small same-origin page map for an agent
fetchkit fetch https://example.com --content-focus agent --crawl --max-pages 5
# Optional JS/DOM rendering for simple SPAs/docs (requires render-rakers feature)
fetchkit fetch https://example.com/app --render-rakers
# Show full documentation
fetchkit --llmtxt
```
Default output is markdown with YAML frontmatter:
```markdown
---
url: https://example.com
status_code: 200
source_content_type: text/html; charset=UTF-8
source_size: 1256
quality_score: 1.00
extraction_method: "full"
---
# Example Domain
This domain is for use in illustrative examples in documents...
```
JSON output (`-o json`):
```json
{
"url": "https://example.com",
"status_code": 200,
"content_type": "text/html",
"size": 1256,
"format": "markdown",
"content": "# Example Domain\n\nThis domain is for use in illustrative examples..."
}
```
## MCP Server
Run as a Model Context Protocol server:
```bash
fetchkit mcp
# Hardened profile for cluster/data-plane use
fetchkit mcp --hardened
```
Exposes `fetchkit` tool over JSON-RPC 2.0 stdio transport. Returns markdown with frontmatter (same format as CLI). Compatible with Claude Desktop and other MCP clients.
## Library Usage
Add to `Cargo.toml`:
```toml
[dependencies]
fetchkit = "0.2"
```
Optional rendered fetching:
```toml
[dependencies]
fetchkit = { version = "0.2", features = ["render-rakers"] }
```
`render-rakers` is not enabled by default. It is lightweight partial rendering:
inline JavaScript can update the DOM before markdown/text conversion, but it is
not a full browser engine. FetchKit blocks rakers-initiated subresource network
access in this mode; the initial page still uses FetchKit's normal URL, DNS,
proxy, timeout, and size policies.
### Basic Fetch
```rust
use fetchkit::{fetch, FetchRequest};
#[tokio::main]
async fn main() {
let request = FetchRequest::new("https://example.com").as_markdown();
let response = fetch(request).await.unwrap();
println!("{}", response.content.unwrap_or_default());
}
```
### With Tool Builder
```rust
use fetchkit::{FetchRequest, ToolBuilder};
let tool = ToolBuilder::new()
.enable_markdown(true)
.enable_text(false)
.user_agent("MyBot/1.0")
.allow_prefix("https://docs.example.com")
.block_prefix("https://internal.example.com")
.build();
let request = FetchRequest::new("https://example.com");
let response = tool.execute(request).await.unwrap();
```
### Toolkit Contract Surface
```rust
use fetchkit::ToolBuilder;
let builder = ToolBuilder::new().enable_save_to_file(true);
let tool = builder.build();
assert_eq!(tool.name(), "web_fetch");
assert_eq!(tool.display_name(), "Web Fetch");
let definition = builder.build_tool_definition();
let mut service = builder.build_service();
```
### Hardened Tool Profile
```rust
use fetchkit::Tool;
let tool = Tool::builder()
.hardened()
.allow_prefix("https://docs.example.com")
.build();
```
## Python Bindings
```bash
pip install fetchkit
```
```python
from fetchkit_py import fetch, FetchRequest, FetchkitTool
# Simple fetch
response = fetch("https://example.com", as_markdown=True)
print(response.content)
# With configuration
tool = FetchkitTool(
enable_markdown=True,
user_agent="MyBot/1.0",
allow_prefixes=["https://docs.example.com"]
)
response = tool.fetch("https://example.com")
```
## Request Fields
| `url` | string | URL to fetch (required, `http://` or `https://`) |
| `method` | enum? | `GET` (default) or `HEAD` |
| `as_markdown` | bool? | Convert HTML to markdown |
| `as_text` | bool? | Convert HTML to plain text |
| `save_to_file` | string? | Non-blank destination path; validated by `FileSaver` before fetching |
| `content_focus` | string? | `"full"`/unset returns everything; `"main"` strips semantic boilerplate; `"readable"` selects article-like content; `"agent"` selects the best low-noise strategy for AI agents |
| `crawl` | bool? | Fetch the seed URL, then discover and fetch bounded same-origin pages |
| `max_pages` | int? | Maximum crawl pages, including the seed; default 5, max 20 |
| `if_none_match` | string? | ETag for conditional `If-None-Match` |
| `if_modified_since` | string? | Timestamp for conditional `If-Modified-Since` |
| `render` | string? | `"rakers"` to opt into rendered fetch when enabled |
## Response Fields
| `url` | string | Fetched URL |
| `status_code` | int | HTTP status code |
| `content_type` | string? | Content-Type header |
| `size` | int? | Content size in bytes |
| `last_modified` | string? | Last-Modified header |
| `etag` | string? | ETag header (use for next conditional request) |
| `filename` | string? | From Content-Disposition |
| `format` | string? | `"markdown"`, `"text"`, `"raw"`, or a fetcher-specific format |
| `content` | string? | Page content |
| `truncated` | bool? | True if content was cut off |
| `method` | string? | `"HEAD"` for HEAD requests |
| `error` | string? | Error message if failed |
| `saved_path` | string? | Filesystem path when `save_to_file` succeeded |
| `bytes_written` | int? | Bytes saved to file |
| `metadata` | object? | Structured `PageMetadata` (title, description, links, headings, extraction method, …) |
| `quality` | object? | Agent-facing `PageQuality` (score, warnings, link density, suggested next action) |
| `crawl` | object? | Bounded crawl discovery result with visited page summaries |
| `word_count` | int? | Word count of returned content |
| `redirect_chain` | string[] | URLs visited during redirects (empty if none) |
| `is_paywall` | bool? | Heuristic paywall signal (soft, not guaranteed) |
| `rendered_by` | string? | Rendering backend used before conversion, e.g. `"rakers"` |
## Error Handling
Errors are returned in the `error` field:
- `InvalidUrl` - Malformed URL
- `UrlBlocked` - URL blocked by filter
- `NetworkError` - Connection failed
- `Timeout` - Request timed out
- `HttpError` - 4xx/5xx response
- `ContentError` - Failed to read body
- `BinaryContent` - Binary content has no registered processor
## Security
Fetchkit blocks connections to private/reserved IP ranges by default, preventing SSRF attacks when used in server-side or AI agent contexts.
**Blocked by default:** loopback, private networks (10.x, 172.16-31.x, 192.168.x), link-local (169.254.x including cloud metadata), IPv6 equivalents, multicast, and other reserved ranges.
```rust
// Default: private IPs blocked (safe for production)
let tool = Tool::default();
// Explicit opt-out for local development only
let tool = Tool::builder()
.block_private_ips(false)
.build();
```
DNS pinning prevents DNS rebinding attacks. IPv6-mapped IPv4 addresses are canonicalized before validation.
Redirects are followed manually in the default fetcher so each hop is revalidated against scheme and DNS policy. Allow/block prefixes are matched against parsed URLs rather than raw strings, which prevents lookalike host overmatches such as `allowed.example.com.evil.test`.
Proxy environment variables are ignored by default. Use the hardened profile for cluster-facing deployments and opt in with `ToolBuilder::respect_proxy_env(true)` only when it is part of an intentional egress design.
See the [`knowledge/security/threat-model.md`](knowledge/security/threat-model.md) concept for the full threat model.
See [`docs/hardening.md`](docs/hardening.md) for deployment guidance.
## Configuration
### Timeouts And Limits
- **First-byte**: 1 second (connect + initial response)
- **Body**: 30 seconds total
- **Body size**: 10 MB decompressed content by default
Partial content is returned on body timeout or body-size limit with `truncated: true`.
### PDF And Binary Content
When Markdown is requested, text-based PDFs are downloaded within the document
size limit (25 MB default) and converted to Markdown. PDF parsing runs locally; no OCR service or
additional network request is used. Scanned/image-only PDFs report `use_ocr` as the
suggested next action.
Office documents are converted the same way, locally: Word headings, lists and
tables, one section per PowerPoint slide, and one Markdown table per spreadsheet
sheet (DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS). Legacy DOC and PPT files are not
converted by the built-in local converters; route them to a remote or custom
converter if needed.
Other binary content returns metadata only:
- Images, audio, video, fonts
- Archives (zip, tar, rar, 7z)
- Legacy Word and PowerPoint files (DOC, PPT)
### HTML Conversion
HTML is automatically converted to markdown:
- Headers: `h1-h6` → `#` to `######`
- Lists: Proper nesting with 2-space indent
- Code: Language-aware, collision-safe fences and inline backticks
- Links/images: Titles, relative-URL resolution, and highest-resolution `srcset`
- Tables: Valid Markdown with formatted cells and escaped pipes
- Rich content: Footnotes, LaTeX math, callouts, figures, highlights,
strikethrough, details, and task checkboxes
- Strips: document head, scripts, styles, templates, iframes, SVGs
- Adds a bounded [Agent resources](docs/agent-discoverability.md) navigation appendix
when discoverable resources are available
## Ecosystem
fetchkit is part of [Everruns](https://everruns.com/projects/), a free, non-commercial
community ecosystem of open-source AI projects by
[Mykhailo Chalyi](https://chaliy.name) and contributors. Questions, ideas and
research discussion are welcome on [Discord](https://everruns.com/discord).
## License
MIT. See [Third-Party Notices](THIRD_PARTY_NOTICES.md) for adapted components.