robotxt 0.6.1

The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
Documentation 
{
  "git": {
    "sha1": "4e876ccee9aca69b036fc3af1d0a3c6f0a73c003"
  },
  "path_in_vcs": "exclusion"
}