robotparser-rs
robots.txt parser for Rust.
Installation
Add it to your Cargo.toml
:
[]
= "0.10"
Add extern crate robotparser
to your crate root and your're good to go!
Examples
extern crate robotparser;
use RobotFileParser;
License
This work is released under the MIT license. A copy of the license is provided in the LICENSE file.