Crate robotparser_fork
source · [−]Expand description
robots.txt parser for Rust
The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt
Installation
Add it to your Cargo.toml
:
[dependencies]
robotparser = "0.10"
Add extern crate robotparser
to your crate root and your’re good to go!
Examples
ⓘ
extern crate robotparser;
use robotparser::RobotFileParser;
fn main() {
let parser = RobotFileParser::new("http://www.python.org/robots.txt");
parser.read();
assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
}
Structs
robots.txt file parser