Crate robots_parser

A robots.txt parser and applicability checker for Rust

The Parser is implemented and tested after http://www.robotstxt.org/norobots-rfc.txt

Usage

Add it to your Cargo.toml:

[dependencies]
robots-parser = "0.10"

Example

use robots::RobotsParser;
use url::Url;

fn main() {
    let parsed = RobotsParser::parse_url(Url::new("https://www.google.com/robots.txt"))?;
    assert!(parsed.can_fetch("*", "https://www.google.com/search/about"));
}

Structs

RobotsParser

Enums

Robots
Rule