Crate robotparser_rs 

A faithful Rust port of Python’s urllib.robotparser.

Port target: the standard library urllib/robotparser.py (CPython 3.13). Behaviour is verified against the reference module itself via differential testing.

Unlike the existing robotparser crate (which follows the 1996 norobots-rfc draft), this mirrors what CPython actually does today: a prefix-match, first-match algorithm (no */$ wildcards, no longest-match precedence), with CPython’s exact path normalization, parse state machine, and user-agent substring matching.

The network half of RobotFileParser (read() / urlopen) is out of scope — fetch the bytes yourself and feed the lines to RobotFileParser::parse, exactly as CPython’s read() does.

use robotparser_rs::RobotFileParser;

let mut rp = RobotFileParser::new("http://example.com/robots.txt");
rp.parse(&["User-agent: *", "Disallow: /private", "Allow: /private/ok"]);
assert_eq!(rp.can_fetch("MyBot", "http://example.com/public"), Ok(true));
assert_eq!(rp.can_fetch("MyBot", "http://example.com/private/x"), Ok(false));

Structs

RequestRate

A Request-rate directive: requests per seconds. Port of the RequestRate namedtuple.

RobotFileParser

Reads, parses and answers questions about a single robots.txt file. Port of RobotFileParser.