Struct Robots 

pub struct Robots { /* private fields */ }

Available on crate feature robots only.

How a crawl fetches and obeys robots.txt, see Crawler::with_robots.

robots.txt is fetched once per origin (scheme, host and port) before its first request, and handled as RFC 9309 describes:

• Its rules for user agent decide what is crawled; disallowed requests are skipped, and Crawl-delay slows the host down.
• A 4xx other than 429, or too many redirects, allows everything.
• A 5xx, a 429 or a network failure disallows everything while robots.txt is retried with backoff. Once the retries are spent, the origin’s requests are skipped.
• Rules older than the maximum age are fetched again; until then, and if fetching fails, the old rules apply.

robots.txt is fetched with the crawl’s backend unless with_backend sets another, e.g. an HTTP client for a browser crawl. with_sitemaps also queues the sitemaps it lists.

use std::time::Duration;

use spire_core::backend::Noop;
use spire_core::crawler::Robots;

let robots = Robots::new("MyBot")
    .with_backend(Noop)
    .with_max_age(Duration::from_secs(3600));

Implementations

impl Robots

pub fn new(user_agent: impl Into<Arc<str>>) -> Self

Obeys robots.txt for user_agent, fetching it with the crawl’s backend, keeping it for 24 hours and retrying an unreachable one 3 times.

pub fn with_backend<B: Backend>(self, backend: B) -> Self

Fetches robots.txt with backend instead of the crawl’s.

pub const fn with_max_age(self, age: Duration) -> Self

Fetches robots.txt again once its rules are older than age. RFC 9309 asks for at most 24 hours.

pub const fn with_unreachable_retries(self, retries: u32) -> Self

Retries an unreachable robots.txt retries times, 30 seconds apart and doubling, before skipping the origin’s requests.

pub fn with_sitemaps(self, tag: impl Into<Tag>) -> Self

Queues the sitemaps an origin’s robots.txt lists in Sitemap: lines, tagged tag, the first time it is fetched.

They go through the frontier like any other seed, so its scope, deduplication and depth limits apply. Route the tag to a handler that parses them, e.g. FollowSitemap in spire.

Trait Implementations

impl Clone for Robots

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Robots

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&str> for Robots

fn from(user_agent: &str) -> Self

Converts to this type from the input type.

impl From<String> for Robots

fn from(user_agent: String) -> Self

Converts to this type from the input type.