Struct RobotsTxtManager 

pub struct RobotsTxtManager { /* private fields */ }

Fetches, parses, and caches robots.txt rules per domain.

You do not construct this directly; the CrawlerEngine creates one when robots_txt_obey is enabled. The manager maintains an internal cache so that each domain’s robots.txt is fetched at most once per crawl run.

Implementations

impl RobotsTxtManager

pub fn new() -> Self

Creates a new robots.txt manager with an empty cache. Rules will be fetched on demand the first time a URL on a new domain is checked.

pub async fn can_fetch( &mut self, url: &str, sid: &str, session_manager: &SessionManager, ) -> bool

Returns true if the URL is allowed by the domain’s robots.txt rules.

The method extracts the domain from the URL, fetches and parses robots.txt if it hasn’t been cached yet, and checks whether the URL’s path matches any Disallow directive. URLs that cannot be parsed (e.g., invalid format) are considered allowed.

pub async fn get_crawl_delay( &mut self, url: &str, sid: &str, session_manager: &SessionManager, ) -> Option<f64>

Returns the crawl-delay specified in the domain’s robots.txt, if any. The delay is in seconds and comes from the Crawl-delay directive under User-agent: *. Returns None if no delay is specified or if the domain’s robots.txt could not be fetched.

pub async fn prefetch( &mut self, urls: &[String], sid: &str, session_manager: &SessionManager, )

Pre-fetches robots.txt for all unique domains in the given URLs.

The engine calls this before the crawl loop starts so that the first batch of requests is not delayed by on-demand robots.txt lookups. Duplicate domains are deduplicated internally.

Trait Implementations

impl Default for RobotsTxtManager

fn default() -> Self

Returns the “default value” for a type.