[−][src]Crate robots_txt
Standard
A Standard for Robot Exclusion
- User-agent
- Disallow
Additions
- Allow
- Crawl-delay
- Request-rate
- Sitemap
- Host
Re-exports
pub use self::parts::Robots; |
Modules
builder | |
matcher | |
parse | |
parts |
A Standard for Robot Exclusion
pub use self::parts::Robots; |
builder | |
matcher | |
parse | |
parts |