robotparser 0.4.0

robots.txt parser for Rust
docs.rs failed to build robotparser-0.4.0
Please check the build logs for more information.
See Builds for ideas on how to fix a failed build, or Metadata for how to configure docs.rs builds.
If you believe this is docs.rs' fault, open an issue.
Visit the last successful build: robotparser-0.11.0

robotparser-rs

Build Status Build status Coverage Status Crates.io

robots.txt parser for Rust.

Installation

Add it to your Cargo.toml:

[dependencies]
robotparser = "*"

Add extern crate robotparser to your crate root and your're good to go!

Examples

extern crate robotparser;

use robotparser::RobotFileParser;

fn main() {
    let parser = RobotFileParser::new("http://www.python.org/robots.txt");
    parser.read();
    assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
}

License

This work is released under the MIT license. A copy of the license is provided in the LICENSE file.