website_crawler 0.1.1

crawl all urls on a website async & sync
1
web: ROCKET_PORT=$PORT CRAWL_URL=$CRAWL_URL ./target/release/crawler