RUSTScrapling

A high-performance Rust port of Scrapling — the modern web scraping framework built by web scrapers, for web scrapers.
Parse HTML with CSS selectors, survive page redesigns with adaptive selectors, and crawl entire sites with async concurrency — from a single Rust crate.
What is this?
Web scraping in three sentences:
- You point it at HTML — a string, a URL, or a whole site — and query it with plain CSS selectors (
page.css(".price")). - When a site redesigns and your selector breaks, adaptive mode finds the element again by similarity to what it looked like before — no code change needed.
- When one page isn't enough, the spider engine crawls whole sites concurrently with rate limiting, deduplication, robots.txt compliance, and pause/resume checkpoints.
It's a faithful Rust port of Python's Scrapling (same concepts, same API names), which means: ~5× faster extraction, ~4× less memory, and a single native binary — no interpreter, no virtualenv, no cold start. Benchmarks below.
| Layer | What it gives you | Key types |
|---|---|---|
| Parser | CSS selectors, DOM navigation, regex extraction, adaptive relocation | Selector, Selectors |
| Fetchers | Async HTTP with retries, stealth headers, charset-aware decoding, proxy rotation | Fetcher, Response |
| Spiders | Concurrent crawling: rules, sitemaps, dedup, checkpoints | Spider, CrawlSpider, SitemapSpider, CrawlerEngine |
| Core | Rich text values, attribute maps, SQLite element storage | TextHandler, SqliteStorage |
Each layer works on its own — use just the parser, or the whole stack.
Table of Contents
- Quick Start
- How fast is it?
- Using it in AI agent workflows
- Usage Guide
- CLI
- Architecture
- Testing
- Contributing
- License
- Credits
Quick Start
Add to your Cargo.toml:
[]
= "0.2"
= { = "1", = ["full"] }
= "1" # items are serde_json::Value
= "0.1" # only needed when implementing the Spider trait
Or track the latest development version straight from git:
= { = "https://github.com/Liohtml/RUSTScrapling" }
Requirements: Rust 1.88+ (edition 2021)
Thirty seconds to your first scrape:
use ;
async
Crawl a whole site from its sitemap in one builder chain:
use ;
use SessionManager;
use Arc;
async
How fast is it?
Measured head-to-head against the original Python Scrapling (v0.4.10, lxml-backed) on the same machine, same document, same selectors, same extraction work: a 616 KB e-commerce page with 1,000 product cards; each run parses the page and extracts title, price, and link from every card. Median of 30 runs, single-threaded.
| Workload | Python Scrapling 0.4.10 | RUSTScrapling (release) | Speedup |
|---|---|---|---|
| Parse + extract 3,000 fields from 1,000 products | 105.7 ms | 19.2 ms | ~5.5× |
| Parse only (build the DOM) | 15.6 ms | 15.7 ms | ~1× |
| Peak process memory for the same job¹ | 37 MB | 9 MB | ~4× |
¹ Peak RSS of a single parse+extract run: Python via resource.getrusage(RUSAGE_SELF).ru_maxrss, Rust via VmHWM in /proc/self/status. The timing scripts below don't measure memory.
What the numbers mean, honestly:
- Raw HTML parsing is a tie — lxml's C parser is excellent, and html5ever matches it. The Rust win is everything after parsing: selector matching, DOM traversal, and text extraction have no interpreter overhead, so end-to-end extraction is ~5.5× faster.
- Memory scales the same way: ~9 MB peak vs ~37 MB for one page. Crawling with 8 concurrent workers, that gap compounds.
- Concurrency is where Rust pulls furthest ahead in practice: the spider engine runs on
tokio— thousands of in-flight requests on a few OS threads, no GIL, no multiprocessing serialization overhead when fanning work out. - Deployment:
cargo build --releaseproduces one native binary. No Python runtime, no dependency resolution at deploy time, millisecond cold starts — which matters for serverless scrapers and agent tools (next section).
Reproduce it yourself — the fixture generator and both benchmark scripts are in the repo:
# from the repo root:
Using it in AI agent workflows
Scraping is one of the most common tools an LLM agent calls — and one of the most punishing: it runs on every task, latency lands inside the agent loop, and a broken selector silently poisons downstream reasoning. RUSTScrapling is a good fit for exactly those reasons:
- Latency lives in your agent loop. 19 ms instead of 106 ms per page extraction matters when an agent chains fetch → extract → reason several times per task.
- Adaptive selectors reduce silent breakage. When a target site redesigns,
css_adaptiverelocates the element by similarity instead of returning nothing — your agent's tool keeps working instead of feeding it empty results. - Deterministic, structured output. Selectors + JSON out, no LLM tokens spent parsing HTML. Feed the model data, not markup.
- One native binary. Trivially shippable as a sandboxed tool next to your agent runtime — no Python environment in the container.
Pattern 1 — expose it as a tool (function calling / MCP)
Wrap fetch + extract in one function with a JSON contract, then register it as a tool in your agent framework (Anthropic tool use, an MCP server, LangChain, etc.):
use ;
/// Tool: fetch a URL and extract elements by CSS selector.
/// Input: { "url": "...", "selector": "..." }
/// Output: [ { "tag": "...", "text": "...", "attrs": {...} } ]
pub async
A matching tool definition for the model:
Pattern 2 — the CLI as a zero-code agent tool
Agents that can run shell commands can use RUSTScrapling with no integration work at all — the CLI emits JSON:
Allow-list that one binary in your agent's sandbox and you have a fast, memory-safe scraping tool with no network stack inside the agent process itself.
Pattern 3 — self-healing extraction for long-running agents
Autonomous agents monitoring sites for weeks can't page a human when a selector dies. Save the element once, then let relocation absorb redesigns:
use SqliteStorage;
use DEFAULT_RELOCATION_PERCENTAGE;
let store = new?;
// css_adaptive = normal CSS first; if the selector stops matching,
// find the element again by similarity to its saved snapshot.
let prices = page.css_adaptive?;
The agent's extraction tool now degrades gracefully: redesign → relocation → same JSON output, and the refreshed snapshot tracks the new page structure.
Pattern 4 — bulk collection for RAG pipelines
Use SitemapSpider to turn a whole site into clean JSONL for chunking and embedding — the quick-start example above is exactly that: robots.txt → sitemap index → filtered URLs → structured items, with dedup, rate limiting, and resumable checkpoints for large corpora built in.
Usage Guide
Parsing HTML
Create a Selector from any HTML string. It wraps the parsed DOM tree and provides the full query API.
use Selector;
let page = from_html;
// With a base URL (enables urljoin for relative links)
let page = from_html_with_url;
let absolute = page.urljoin; // "https://example.com/about"
CSS Selectors
Full CSS3 selector support powered by the scraper crate:
let page = from_html;
let items = page.css; // by class
let main = page.css; // by ID
let priced = page.css; // by attribute
let links = page.css; // compound
The result is a Selectors collection with batch operations:
let items = page.css;
let first = &items;
for item in &items
let special = items.filter;
let target = items.search;
let names = page.css.css; // chain queries
let all_text: = items.getall;
Parsel-style ::text and ::attr(name) pseudo-elements extract values in
one call instead of two:
let titles = page.css_getall; // recursive text per match
let hrefs = page.css_getall; // attribute values
let first = page.css_get; // first value only
Adaptive Selectors (survive redesigns)
Save an element's "fingerprint" (tag, attributes, text, position, parent/sibling context) to SQLite. When the selector later stops matching — because the site changed — relocate/css_adaptive scores every element in the new page against the snapshot and returns the best match above a similarity threshold:
use SqliteStorage;
use ;
let store = new?;
// First run: selector works, element gets saved under "product-card".
let page = from_html;
page.css_adaptive?;
// After a redesign: same call, selector fails, relocation finds the moved element.
let page = from_html;
let found = page.css_adaptive?;
assert!;
The similarity scoring is a port of upstream Scrapling's algorithm (with a difflib-parity sequence matcher), so behavior matches the Python original.
Text Extraction
TextHandler wraps every text value with regex, JSON, and cleaning methods:
let text = element.text; // direct text
let all = element.get_all_text; // recursive
let cleaned = element.text.strip.to_lowercase.replace_str;
let data = element.text.json?; // parse as JSON
let inner = element.html_content;
let outer = element.outer_html;
DOM Navigation
let products = page.css;
let item = products.first.unwrap;
let list = item.parent.unwrap;
let children = list.children;
let siblings = item.siblings;
let next = item.next;
let prev = item.previous;
let attrs = item.attrib;
let id = attrs.get.unwrap;
let heading = page.find_by_text;
let matched = page.find_by_regex;
Regex Extraction
let price_els = page.css;
let price_el = price_els.first.unwrap;
let prices = price_el.re; // all matches
let first = price_el.re_first; // first match
let all = page.css.re; // batch
Fetching Pages
The Fetcher is an async HTTP client with retries, stealth headers, charset-aware body decoding, and proxy support:
use ;
// Default config: 30s timeout, 3 retries, stealth headers on
let fetcher = new?;
// Custom config via builder
let fetcher = new?;
let resp = fetcher.get.await?;
let resp = fetcher.post.await?;
let page = resp.selector; // Response -> Selector (auto-parses HTML)
println!;
println!;
Response bodies are decoded according to the Content-Type charset (ISO-8859-1, Shift_JIS, windows-1252, …) — no mojibake from non-UTF-8 sites.
Search engines (DuckDuckGo)
DuckDuckGo's HTML endpoint rate-limits automated requests aggressively and on detection returns HTTP 202 with its homepage instead of results. It also wraps result links in a redirect. The fetchers::search helpers handle both:
use ;
let resp = fetcher.get.await?;
if is_duckduckgo_blocked else
Building a Spider
Implement the Spider trait for full control over a crawl:
use ;
use SpiderResponse;
use SessionManager;
use async_trait;
use Arc;
;
async
Spider Configuration Options
| Option | Default | Description |
|---|---|---|
concurrent_requests() |
4 |
Global concurrency limit |
concurrent_requests_per_domain() |
0 |
Per-domain limit (0 = disabled) |
download_delay() |
0.0 |
Seconds between requests |
robots_txt_obey() |
false |
Respect robots.txt (Allow/Disallow with wildcards, Crawl-delay) |
max_blocked_retries() |
3 |
Retry limit for blocked responses |
allowed_domains() |
{} |
Domain whitelist (empty = allow all) |
development_mode() |
false |
Cache responses to disk for dev iteration |
autothrottle_enabled() |
false |
Adaptive per-domain delays (see below) |
autothrottle_start_delay() |
5.0 |
AutoThrottle initial delay in seconds |
autothrottle_max_delay() |
60.0 |
AutoThrottle upper bound in seconds |
autothrottle_target_concurrency() |
1.0 |
Desired in-flight requests per domain |
AutoThrottle
With autothrottle_enabled(), each domain gets an adaptive delay driven
by measured response latency: fast healthy servers are crawled faster
(down to download_delay / robots.txt Crawl-delay as floors), slow ones
slower — and a 429/503 doubles the delay or honors the server's
Retry-After, effective for the very next request. Requests to one domain
are spaced by a per-domain reservation clock, so concurrent tasks queue
behind each other instead of firing at once, and separate domains never
delay each other.
Spider Lifecycle Hooks
| Hook | When |
|---|---|
on_start(resuming) |
Before crawl begins |
on_close() |
After crawl ends |
on_error(request, error) |
When a request fails |
on_scraped_item(item) |
Item pipeline — return None to drop |
is_blocked(response) |
Custom block detection |
Memory note: the scheduler keeps a dedup set of one fingerprint per unique URL visited (~100 MB per ~1M URLs), persisted in checkpoints so paused crawls resume without re-visiting pages. For very large or open-ended crawls, set
allowed_domains()to bound scope.
CrawlSpider, SitemapSpider & LinkExtractor
For the common cases you don't need to implement Spider yourself:
LinkExtractor — declarative URL discovery with regex allow/deny, domain filters (subdomains included), CSS scoping, and a built-in binary-extension deny list (including compound extensions like .tar.gz):
use LinkExtractor;
let extractor = new
.allow.unwrap
.deny.unwrap
.allow_domains // matches api.example.com too
.restrict_css; // only look inside #content
let urls: = extractor.extract; // absolute, deduped, canonicalized
CrawlSpider — follow links matching rules, extract items from every page:
use ;
use Arc;
let spider = builder
.start_urls
.rule
.parse_item
.build;
SitemapSpider — seed a crawl from sitemaps (or robots.txt Sitemap: directives), recurse through sitemap indexes, and dispatch page URLs through rules — see the Quick Start for a complete example.
ShopifySpider — extract every product variant from any Shopify-powered store through its JSON API (collections.json → per-collection products.json), no HTML parsing and no selectors to maintain:
use ShopifySpider;
let spider = builder.build;
// items: { name, price, category, brand, identifier, sku, stock,
// image_url, url, description, old_price, barcode }
CLI
RUSTScrapling includes a command-line tool for quick scraping:
# Fetch a page and extract text
# Extract specific elements with a CSS selector
# Output as HTML or JSON (tag, text, html per element)
# Disable stealth headers
# Extract text content (shorthand)
Architecture
rust_scrapling/
|
|-- core/ # Foundation types
| |-- text_handler.rs # TextHandler: String + regex/json/clean
| |-- text_handlers.rs # TextHandlers: Vec<TextHandler> batch ops
| |-- attributes_handler.rs # AttributesHandler: read-only attr map
| +-- storage.rs # SqliteStorage: adaptive element persistence
|
|-- parser/ # HTML parsing engine
| |-- selector.rs # Selector: element wrapper (CSS, text, nav)
| |-- selectors.rs # Selectors: batch operations
| |-- adaptive.rs # Adaptive relocation: save/retrieve/relocate
| |-- selector_generation.rs # Auto-generate CSS/XPath from DOM position
| +-- translator.rs # ::text and ::attr() pseudo-elements
|
|-- fetchers/ # HTTP layer
| |-- client.rs # Fetcher: async HTTP with retries
| |-- config.rs # FetcherConfig: builder pattern
| |-- encoding.rs # Charset-aware response decoding
| |-- response.rs # Response: auto-parses to Selector
| |-- proxy.rs # ProxyRotator: round-robin proxy cycling
| |-- search.rs # DuckDuckGo helpers (block detection, URL decode)
| +-- constants.rs # User agents, status codes, headers
|
+-- spiders/ # Crawl framework
|-- spider.rs # Spider trait (user-facing API)
|-- engine.rs # CrawlerEngine: async orchestrator
|-- links.rs # LinkExtractor: URL discovery primitive
|-- templates/ # CrawlSpider + CrawlRule, SitemapSpider
|-- request.rs # SpiderRequest: fingerprinting + priority
|-- response.rs # SpiderResponse: parser integration
|-- result.rs # CrawlResult, CrawlStats, ItemList
|-- scheduler.rs # Priority queue with deduplication
|-- session.rs # SessionManager: named HTTP sessions
|-- robots.rs # robots.txt compliance (RFC 9309 matching)
|-- throttle.rs # AutoThrottle: adaptive per-domain delays
|-- cache.rs # Dev-mode response caching
+-- checkpoint.rs # Pause/resume persistence (incl. dedup state)
Design Principles
- Each layer is independent. Use just the parser without fetchers. Use fetchers without spiders. Compose as needed.
- Cheap tree sharing.
SelectorusesRc<Html>to share the parsed tree — child selectors point into the same tree instead of copying it. - Async-first. The fetcher and spider layers are built on
tokiofor high-concurrency crawling. - Scrapy-compatible API names.
css(),text(),re(),re_first(),get(),getall()mirror Scrapy/Parsel conventions.
Testing
CI runs the full suite on Linux, macOS, and Windows, plus clippy, rustfmt, MSRV (1.88), and cargo audit on every push.
Contributing
Contributions are welcome! Fork, branch, write tests, run cargo test && cargo clippy, and open a PR.
Areas for Contribution
- Browser automation — headless Chrome/Playwright integration (like Python Scrapling's
StealthyFetcher/DynamicFetcher) - MCP server — a ready-made Model Context Protocol wrapper around the fetch/extract API
- Interactive shell — REPL for exploring pages
- Performance — more benchmarks, SIMD text processing, zero-copy parsing
- Documentation — more examples, tutorials, API docs
License
Licensed under either of:
- MIT License (LICENSE-MIT or http://opensource.org/licenses/MIT)
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
at your option.
Credits
- Scrapling by Karim Shoair — the original Python framework that inspired this project
- scraper — HTML parsing and CSS selection in Rust
- reqwest — HTTP client
- tokio — Async runtime
- Banner art generated with Z-Image-Turbo on Hugging Face Spaces
Built with Rust. Inspired by Scrapling. Made for scraping.