Struct RobotsTxtCache 

pub struct RobotsTxtCache { /* private fields */ }

Per-domain robots.txt cache.

Thread-safe. Fetches robots.txt once per unique domain, parses it, and caches the rules. Subsequent requests for the same domain return the cached result.

Implementations

impl RobotsTxtCache

pub fn new(client: Arc<HttpClient>, config: &CrawlConfig) -> Self

Create a new cache with the given HTTP client.

pub async fn get( &self, scheme: &str, domain: &str, ) -> (Vec<RobotsRule>, Vec<String>, String)

Get robots.txt rules for a domain.

Returns (rules, sitemap_urls, raw_content). If the domain has no robots.txt, returns empty vectors and an empty string. Fetches from the network on first access for each domain.

pub async fn is_disallowed( &self, scheme: &str, domain: &str, path: &str, ) -> bool

Check if a path is disallowed for the given domain and user-agent.

pub async fn crawl_delay(&self, scheme: &str, domain: &str) -> Option<f64>

Get crawl-delay for a domain (in seconds).

pub async fn sitemaps(&self, scheme: &str, domain: &str) -> Vec<String>

Get all sitemap URLs discovered from robots.txt for a domain.

pub async fn raw_content(&self, scheme: &str, domain: &str) -> String

Get the raw robots.txt content for a domain.