[][src]Crate robots_txt

Standard

A Standard for Robot Exclusion

  • User-agent
  • Disallow

Additions

  • Allow
  • Crawl-delay
  • Request-rate
  • Sitemap
  • Host

Re-exports

pub use self::parts::Robots;

Modules

builder
matcher
parse
parts