[−][src]Crate robots_parser
A robots.txt parser and applicability checker for Rust
The Parser is implemented and tested after http://www.robotstxt.org/norobots-rfc.txt
Usage
Add it to your Cargo.toml
:
[dependencies]
robots-parser = "0.10"
Example
ⓘThis example is not tested
use robots::RobotsParser; use url::Url; fn main() { let parsed = RobotsParser::parse_url(Url::new("https://www.google.com/robots.txt"))?; assert!(parsed.can_fetch("*", "https://www.google.com/search/about")); }
Structs
RobotsParser |
Enums
Robots | |
Rule |