Crate robots_txt

Standard

A Standard for Robot Exclusion

• User-agent
• Disallow

Additions

• Allow
• Crawl-delay
• Request-rate
• Sitemap
• Host

Re-exports

pub use self::parts::Robots;

Modules

builder

matcher

parse

parts