Crate robotxt 

robotxt

Check out other spire projects here.

The implementation of the robots.txt (or URL exclusion) protocol in the Rust programming language, following RFC 9309, with the support of the crawl-delay and sitemap extensions.

Features

• parser to enable robotxt::{Robots}. Enabled by default.
• builder to enable robotxt::{RobotsBuilder, GroupBuilder}. Enabled by default.
• serde to enable serde::{Deserialize, Serialize} implementations, allowing the caching of parsed rules.

Examples

• parse the rules that apply to the crawler:

use robotxt::Robots;

let txt = b"
  User-Agent: foobot
  Disallow: *
  Allow: /example/
  Disallow: /example/nope.txt
";

let r = Robots::parse(txt, "foobot");
assert!(r.is_allowed("/example/yeah.txt"));
assert!(!r.is_allowed("/example/nope.txt"));
assert!(!r.is_allowed("/invalid/path.txt"));

• match the crawler’s name by prefix instead of exactly, so a foobot group also applies to the foobot-images crawler:

use robotxt::{Agent, Robots};

let txt = b"
  User-Agent: foobot
  Disallow: /
";

let r = Robots::parse(txt, Agent::Prefix("foobot-images"));
assert!(!r.is_allowed("/"));

• build a new robots.txt file:

use std::time::Duration;
use robotxt::RobotsBuilder;

let txt = RobotsBuilder::new()
    .header("Robots.txt: Start")
    .group(["foobot"], |u| {
        u.crawl_delay(Duration::from_secs(5))
            .header("Rules for Foobot: Start")
            .allow("/example/yeah.txt")
            .disallow("/example/nope.txt")
            .footer("Rules for Foobot: End")
    })
    .group(["barbot", "nombot"], |u| {
        u.crawl_delay(Duration::from_secs(2))
            .disallow("/example/yeah.txt")
            .disallow("/example/nope.txt")
    })
    .sitemap("https://example.com/sitemap_1.xml".parse().unwrap())
    .sitemap("https://example.com/sitemap_2.xml".parse().unwrap())
    .footer("Robots.txt: End");

println!("{txt}");

Links

• Request for Comments: 9309 on RFC-Editor.com
• Introduction to Robots.txt on Google.com
• How Google interprets Robots.txt on Google.com
• What is Robots.txt file on Moz.com

Notes

• Only User-Agent, Allow, Disallow, Crawl-Delay and Sitemap are understood. Other lines, e.g. the non-standard Host or Clean-Param directives, are ignored.

Re-exports

pub use url;

Structs

GroupBuilderbuilder

The builder of a single User-Agent group.

Robotsparser

The rules of a robots.txt file that apply to a single crawler.

RobotsBuilderbuilder

The builder of robots.txt files.

Ruleparser

A single Allow or Disallow rule of the group that applies to the crawler.

Enums

AccessResultparser

The result of an attempt to fetch the robots.txt file, as described in RFC 9309, section 2.3.1.

Agentparser

The crawler’s name and how it is matched against User-Agent values.

Error

Failure to derive the robots.txt location from a URL.

Constants

ANY_AGENT

The User-Agent value of the group that applies to every crawler without a group of its own.

BYTE_LIMITparser

The maximum number of bytes of a robots.txt file that are parsed.

Functions

robots_url

Returns the location of the robots.txt file that governs the URL.

Type Aliases

Result

A specialized Result type for robotxt operations.