spider 2.48.13

A web crawler and scraper, building blocks for data curation workloads.
1
2

/// robot parser
pub mod robotparser;