url-crawler 0.2.0

Configurable parallel web crawler, designed to crawl for content
Documentation