robotparser-rs 0.1.0

A faithful Rust port of Python's urllib.robotparser: parse robots.txt and answer can_fetch/crawl_delay/request_rate, byte-for-byte compatible with CPython.
Documentation