Skip to main content

Crate rust_scrapling

Crate rust_scrapling 

Source
Expand description

RUSTScrapling — a Rust port of the Scrapling web-scraping framework.

The crate is organized in three layers, each usable on its own:

  • parser — HTML parsing and data extraction. Selector wraps a parsed DOM node and offers CSS selection (including Parsel-style ::text / ::attr(name) pseudo-elements via Selector::css_get / Selector::css_getall), text extraction, DOM navigation, regex helpers, JSON parsing, and adaptive element relocation — a saved element’s “unique properties” can be used to find it again by similarity after the page layout changes.
  • fetchers — async HTTP. Fetcher is a reqwest-backed client configured through FetcherConfig (timeouts, retries, redirects, single/per-protocol/rotating proxies, response-size caps) that sends browser-like stealth headers by default, and returns a Response with charset-aware body decoding and direct parser integration.
  • spiders — crawling. Implement the Spider trait (or use a ready-made template: CrawlSpider, SitemapSpider, ShopifySpider) and run it with CrawlerEngine. The engine handles concurrency limits (global and per-domain), request deduplication, robots.txt compliance (RFC 9309 matching plus Crawl-delay), AutoThrottle adaptive per-domain delays, blocked-response retries, checkpoints for pause/resume, and a development-mode response cache for offline iteration. Results come back as a CrawlResult with live stats and an ItemList exportable to CSV, JSON, JSON Lines, or XML.

§Quickstart: parsing HTML

use rust_scrapling::Selector;

let html = r#"<html><body>
  <ul>
    <li class="item"><a href="/p/1">Widget</a> <span class="price">9.99</span></li>
    <li class="item"><a href="/p/2">Gadget</a> <span class="price">19.99</span></li>
  </ul>
</body></html>"#;

let page = Selector::from_html(html);

// Parsel-style `::text` / `::attr()` pseudo-elements:
let names: Vec<String> = page
    .css_getall("li.item a::text")
    .into_iter()
    .map(|t| t.into_string())
    .collect();
assert_eq!(names, ["Widget", "Gadget"]);

let first_href = page.css_get("li.item a::attr(href)").unwrap();
assert_eq!(first_href.as_str(), "/p/1");

// Plain CSS queries return `Selectors` for further navigation:
let prices = page.css(".price");
assert_eq!(prices.len(), 2);
assert_eq!(prices[0].text().as_str(), "9.99");

§Fetching a page

use rust_scrapling::{Fetcher, FetcherConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = FetcherConfig::builder()
        .timeout(20)
        .stealth(true) // browser-like headers (the default)
        .build();
    let fetcher = Fetcher::new(config)?;

    let response = fetcher.get("https://example.com").await?;
    println!("status: {}", response.status());
    if let Some(title) = response.selector().css_get("title::text") {
        println!("title: {}", title);
    }
    Ok(())
}

§A minimal spider

use async_trait::async_trait;
use rust_scrapling::spiders::response::SpiderResponse;
use rust_scrapling::spiders::session::SessionManager;
use rust_scrapling::{CrawlerEngine, FetcherConfig, Spider, SpiderRequest};
use std::sync::Arc;

struct QuotesSpider;

#[async_trait]
impl Spider for QuotesSpider {
    fn name(&self) -> &str {
        "quotes"
    }

    fn start_urls(&self) -> Vec<String> {
        vec!["https://quotes.toscrape.com/".to_string()]
    }

    async fn parse(
        &self,
        response: SpiderResponse,
    ) -> (Vec<serde_json::Value>, Vec<SpiderRequest>) {
        let items = response
            .css_getall(".quote .text::text")
            .into_iter()
            .map(|text| serde_json::json!({ "quote": text.as_str() }))
            .collect();
        (items, Vec::new()) // no follow-up requests
    }
}

#[tokio::main]
async fn main() {
    let sessions = SessionManager::new(FetcherConfig::default());
    let engine = CrawlerEngine::new(Arc::new(QuotesSpider), sessions, None)
        .expect("HTTP client should build");
    let result = engine.crawl().await;
    println!("scraped {} items", result.items.len());
}

§Feature highlights

Re-exports§

pub use fetchers::client::Fetcher;
pub use fetchers::client::FetcherError;
pub use fetchers::config::FetcherConfig;
pub use fetchers::response::Response;
pub use parser::Selector;
pub use parser::Selectors;
pub use spiders::engine::CrawlerEngine;
pub use spiders::links::LinkExtractor;
pub use spiders::request::SpiderRequest;
pub use spiders::result::CrawlResult;
pub use spiders::result::CrawlStats;
pub use spiders::result::ItemList;
pub use spiders::session::SessionError;
pub use spiders::session::SessionManager;
pub use spiders::spider::Spider;
pub use spiders::templates::CrawlRule;
pub use spiders::templates::CrawlSpider;
pub use spiders::templates::CsvFeedSpider;
pub use spiders::templates::ShopifySpider;
pub use spiders::templates::SiteToMarkdownSpider;
pub use spiders::templates::SitemapSpider;
pub use spiders::templates::XmlFeedSpider;

Modules§

core
Shared value types used across the parser and spiders: string wrappers with scraping helpers (TextHandler, TextHandlers), a read-only attribute map (AttributesHandler), and the SQLite storage backing adaptive element relocation (storage::SqliteStorage).
fetchers
Async HTTP fetching: a reqwest-backed client (client::Fetcher) with stealth headers, retries, proxy support (single, per-protocol, and rotating), charset-aware body decoding, and response wrappers that plug straight into the parser.
parser
HTML parsing and data extraction: CSS selection with ::text/::attr() pseudo-elements, DOM navigation, selector generation, and adaptive element relocation.
spiders
Crawling: the spider::Spider trait, the engine::CrawlerEngine that runs it, and the supporting machinery — scheduling with request dedup, robots.txt compliance, AutoThrottle, checkpoints for pause/resume, a development-mode response cache, link extraction, and crawl results/exports.