Crate robotparser [−] [src]
robots.txt parser for Rust
The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt
Installation
Add it to your Cargo.toml
:
[dependencies]
robotparser = "0.10"
Add extern crate robotparser
to your crate root and your're good to go!
Examples
⚠Be careful when using this code, it's not being tested!
extern crate robotparser; use robotparser::RobotFileParser; fn main() { let parser = RobotFileParser::new("http://www.python.org/robots.txt"); parser.read(); assert!(parser.can_fetch("*", "http://www.python.org/robots.txt")); }
Structs
RequestRate | |
RobotFileParser |
robots.txt file parser |