spider 2.48.22

A web crawler and scraper, building blocks for data curation workloads.
1
2

/// robot parser
pub mod robotparser;