rust_scrapling 0.4.1

Modern web scraping: spiders, CSS selectors, robots.txt/AutoThrottle politeness, and LLM/RAG-ready Markdown extraction - a Rust port of Scrapling
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

//! Crawling: the [`spider::Spider`] trait, the [`engine::CrawlerEngine`]
//! that runs it, and the supporting machinery — scheduling with request
//! dedup, robots.txt compliance, AutoThrottle, checkpoints for
//! pause/resume, a development-mode response cache, link extraction, and
//! crawl results/exports.

pub mod cache;
pub mod checkpoint;
pub mod engine;
pub mod links;
pub mod request;
pub mod response;
pub mod result;
pub mod robots;
pub mod scheduler;
pub mod session;
pub mod spider;
pub mod templates;
pub mod throttle;