url-crawler 0.2.1

Configurable parallel web crawler, designed to crawl for content
Documentation