codoseo-crawler 0.0.1

The CodoSEO crawler: fetching, robots.txt, sitemaps and streaming field extraction.
Documentation 
1

<html><head><meta name="robots" content="noindex, follow"><meta name="ROBOTS" content="nosnippet"><title>x</title></head><body>hi</body></html>