robotxt 0.1.0

xwde: robotxt

The implementation of the robots.txt protocol (or URL exclusion protocol) in Rust programming language with the support of crawl-delay, sitemap and universal * match extensions (according to the RFC specification).

Examples

• parse the user-agent in the provided robots.txt file:

use robotxt::Robots;

fn main() {
    let txt = r#"
      User-Agent: foobot
      Allow: /example/
      Disallow: /example/nope.txt
    "#.as_bytes();
    
    let r = Robots::from_slice(txt, "foobot");
    assert!(r.is_match("/example/yeah.txt"));
    assert!(!r.is_match("/example/nope.txt"));
}

• build the new robots.txt file from provided directives:

Note : the builder is not yet implemented.

Links

• Request for Comments: 9309 on RFC-Editor.com
• Introduction to Robots.txt on Google.com
• How Google interprets Robots.txt on Google.com
• What is Robots.txt file on Moz.com

Notes

The parser is based on Smerity/texting_robots.