crawly 0.1.2

A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
Documentation 
1
2
3
4
5
6
7

.idea
.vscode
**/*.rs.bk
/*.data
/*.gz
*.local
/target