# 🦀 Rust Scraper
[](https://github.com/XaviCode1000/rust-scraper/actions/workflows/ci.yml)
[](https://opensource.org/licenses/MIT)
[](https://www.rust-lang.org/)
[](https://github.com/XaviCode1000/rust-scraper/releases)
Modern web scraper optimized for RAG (Retrieval-Augmented Generation) datasets. Built with **Clean Architecture** for maintainability and production use.
## ✨ Features
### Core
- 📖 **Readability Algorithm** - Extracts clean content like Firefox Reader Mode
- 🌐 **Modern HTTP Client** - reqwest with TLS (rustls), gzip/brotli compression
- 📝 **Multiple Output Formats** - Markdown (with YAML frontmatter), JSON, plain text
- 🔧 **CLI Interface** - Full control via command line arguments
### Production Ready (v0.3.0)
- 🔄 **Retry Logic** - Exponential backoff for transient failures (5xx, timeouts)
- 🎯 **Bounded Concurrency** - Prevents resource exhaustion (3 concurrent for HDD)
- 🎭 **User-Agent Rotation** - 14 modern browsers, weighted selection
- 🛡️ **Type-Safe Errors** - `ScraperError` enum with 14 variants
- 🧪 **Well Tested** - 83 tests (unit, integration, doctests)
### Asset Download
- 🖼️ **Image Download** - Automatic download to `output/images/`
- 📄 **Document Download** - PDF, DOCX, XLSX to `output/documents/`
- 🔍 **MIME Detection** - Automatic classification by extension
- 🔐 **SHA256 Hashing** - Unique filenames, no collisions
## 🚀 Quick Start
```bash
# Clone repository
git clone https://github.com/XaviCode1000/rust-scraper.git
cd rust-scraper
# Build in release mode
cargo build --release
# Run with URL (required)
cargo run --release -- --url "https://example.com"
```
## 🎯 Usage
### Basic Scraping
```bash
# Basic usage (URL is REQUIRED)
cargo run --release -- --url "https://example.com"
# Specify output directory
cargo run --release -- --url "https://example.com" -o ./output
# Choose output format
cargo run --release -- --url "https://example.com" -f json
cargo run --release -- --url "https://example.com" -f text
```
### Asset Downloads
```bash
# Download images only
cargo run --release -- --url "https://example.com" --download-images
# Download documents only
cargo run --release -- --url "https://example.com" --download-documents
# Download both
cargo run --release -- --url "https://example.com" --download-images --download-documents
```
### Advanced Options
```bash
# Full example with all options
cargo run --release -- \
--url "https://example.com/article" \
--selector "article.main" \
--output "./my-output" \
--format markdown \
--download-images \
--download-documents \
--delay-ms 1000 \
--max-pages 10 \
--verbose
```
### CLI Options
| `-u, --url` | **URL to scrape (REQUIRED)** | - |
| `-s, --selector` | CSS selector for content | `body` |
| `-o, --output` | Output directory | `output` |
| `-f, --format` | Output format (markdown/json/text) | `markdown` |
| `--download-images` | Download images to `output/images/` | ❌ |
| `--download-documents` | Download documents to `output/documents/` | ❌ |
| `--delay-ms` | Delay between requests (ms) | `1000` |
| `--max-pages` | Maximum pages to scrape | `10` |
| `-v, --verbose` | Increase verbosity (use multiple times) | - |
```bash
# Get help
cargo run --release -- --help
```
## 📁 Output Structure
### Markdown Output (Default)
```
output/
└── example.com/
└── article/
└── index.md
```
**Markdown file with YAML frontmatter:**
```markdown
---
title: Article Title
url: https://example.com/article
date: "2026-03-08"
author: John Doe
excerpt: A short excerpt
---
# Article Title
Main content here with **markdown** formatting...
```rust
// Code blocks with syntax highlighting
fn main() {
println!("Hello, world!");
}
```
```
### With Asset Downloads
```
output/
├── example.com/
│ └── article/
│ └── index.md
├── images/
│ ├── 027e504eabfc.png
│ ├── 0c2f4f0301fe.png
│ └── e15cbdd2d653.svg
└── documents/
└── 9870371a7a8c.pdf
```
### JSON Output
```bash
cargo run --release -- --url "https://example.com" -f json
```
**output/results.json:**
```json
[
{
"title": "Article Title",
"content": "Main content...",
"url": "https://example.com/article",
"excerpt": "A short excerpt",
"author": "John Doe",
"date": "2026-03-08",
"assets": [
{
"url": "https://example.com/image.png",
"local_path": "output/images/027e504eabfc.png",
"asset_type": "image",
"size": 1024
}
]
}
]
```
## 🏗️ Architecture
This project uses **Clean Architecture** with four layers:
```
┌─────────────────────────────────────┐
│ CLI (main.rs) │ - User interface
└──────────────┬──────────────────────┘
│
┌──────────────▼──────────────────────┐
│ Library (lib.rs) │ - Public API
└──────────────┬──────────────────────┘
│
┌──────────┴──────────┐
│ │
┌───▼──────┐ ┌──────▼────────┐
│ DOMAIN │ │ APPLICATION │
│ (pure) │ │ (use cases) │
└──────────┘ └──────┬────────┘
│
┌──────────┴──────────┐
│ │
┌──────▼──────┐ ┌──────▼──────┐
│INFRASTRUCTURE│ │ ADAPTERS │
└─────────────┘ └─────────────┘
```
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
## 🧪 Testing
```bash
# Run all tests
cargo test
# Run with output
cargo test -- --nocapture
# Run specific test
cargo test test_validate_url
# Run with coverage (requires cargo-tarpaulin)
cargo tarpaulin --out Html
```
**Test Coverage**: 83 tests passing
- 70 unit tests
- 11 doctests
- 2 integration tests
## 📦 Installation (as Library)
Add to your `Cargo.toml`:
```toml
[dependencies]
rust_scraper = "0.3.0"
tokio = { version = "1", features = ["full"] }
anyhow = "1"
```
**Example usage:**
```rust
use rust_scraper::{
create_http_client,
scrape_with_config,
save_results,
validate_and_parse_url,
ScraperConfig,
OutputFormat,
};
use std::path::PathBuf;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Create HTTP client (with retry + user-agent rotation)
let client = create_http_client()?;
// Parse URL
let url = validate_and_parse_url("https://example.com")?;
// Configure scraping
let config = ScraperConfig {
download_images: true,
download_documents: false,
output_dir: PathBuf::from("./output"),
max_file_size: Some(50 * 1024 * 1024), // 50MB
};
// Scrape
let results = scrape_with_config(&client, &url, &config).await?;
// Save results
save_results(&results, &config.output_dir, &OutputFormat::Markdown)?;
Ok(())
}
```
## 🔧 Development
```bash
# Clone and setup
git clone https://github.com/XaviCode1000/rust-scraper.git
cd rust-scraper
# Build in debug mode
cargo build
# Build in release mode (optimized)
cargo build --release
# Run clippy (linting)
cargo clippy -- -D clippy::correctness
# Format code
cargo fmt
# Run all tests
cargo test --all
```
### Release Profile
The project uses aggressive optimizations for production builds:
```toml
[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
panic = "abort"
strip = true
```
## 📋 Requirements
- [Rust](https://rustup.rs/) 1.70+ (edition 2021)
- Linux, macOS, or Windows
```bash
# Check Rust version
rustc --version
# Update if needed
rustup update
```
## 🤝 Contributing
Contributions are welcome! See [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md) for guidelines.
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
## 📄 License
MIT License - see [LICENSE](LICENSE) for details.
## 📚 Documentation
- [Architecture](docs/ARCHITECTURE.md) - Clean Architecture details
- [Changelog](docs/CHANGES.md) - Version history and migration guide
- [CLI Reference](docs/CLI.md) - Complete CLI documentation
- [Contributing](docs/CONTRIBUTING.md) - How to contribute
## 🙏 Acknowledgments
- [legible](https://github.com/relaxnow/legible) - Readability algorithm
- [reqwest](https://github.com/seanmonstar/reqwest) - HTTP client
- [html-to-markdown-rs](https://github.com/ramonh/html-to-markdown) - HTML→Markdown conversion
- [scraper](https://github.com/programble/scraper) - HTML parsing
---
**Made with ❤️ using Rust and Clean Architecture**