siteone-crawler 2.4.1

Website crawler and QA toolkit in Rust for security, performance, SEO, and accessibility audits, offline cloning, markdown export, sitemap generation, cache warming, and CI/CD gating — one dependency-free binary for all major platforms, 10 tools in one.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12

// Engine module - core crawling engine
// (c) Jan Reges <jan.reges@siteone.cz>

pub mod crawler;
pub mod found_url;
pub mod found_urls;
pub mod http_client;
pub mod http_response;
pub mod initiator;
pub mod manager;
pub mod parsed_url;
pub mod robots_txt;