Crate robotparser_fork

robots.txt parser for Rust

The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt

Installation

Add it to your Cargo.toml:

[dependencies]
robotparser = "0.10"

Add extern crate robotparser to your crate root and your’re good to go!

Examples

extern crate robotparser;

use robotparser::RobotFileParser;

fn main() {
    let parser = RobotFileParser::new("http://www.python.org/robots.txt");
    parser.read();
    assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
}

Structs

RequestRate

RobotFileParser

robots.txt file parser