robotparser-fork 0.10.6

robots.txt parser for Rust
Documentation
robotparser-fork-0.10.6 has been yanked.

robotparser-rs

Build Status Build status Coverage Status Crates.io Dependency status

robots.txt parser for Rust.

Installation

Add it to your Cargo.toml:

[dependencies]
robotparser = "0.10"

Add extern crate robotparser_fork to your crate root and your're good to go!

Examples

extern crate robotparser_fork;

use robotparser_fork::RobotFileParser;

fn main() {
    let parser = RobotFileParser::new("http://www.python.org/robots.txt");
    parser.read();
    assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
}

License

This work is released under the MIT license. A copy of the license is provided in the LICENSE file.