url-crawler 0.1.1

Configurable parallel web crawler, designed to crawl for content
Documentation