website_crawler 0.1.2

crawl all urls on a website async & sync
1
web: ROCKET_PORT=$PORT CRAWL_URL=$CRAWL_URL ./target/release/crawler