Struct RobotsFetcher 

pub struct RobotsFetcher { /* private fields */ }

Robots.txt fetcher with RFC 9309 compliance

Implementations

impl RobotsFetcher

pub fn new(config: RobotsConfig) -> Self

Create a new fetcher

pub fn stats(&self) -> Arc<FetchStats>

Get fetch statistics

pub async fn fetch(&self, url: &Url) -> RobotsPolicy

Fetch and parse robots.txt for a URL

pub async fn fetch_for_key(&self, key: &RobotsCacheKey) -> RobotsPolicy

Fetch robots.txt for a cache key

pub async fn fetch_for_key_conditional( &self, key: &RobotsCacheKey, etag: Option<&str>, last_modified: Option<&str>, ) -> RobotsPolicy

Fetch robots.txt with conditional GET support

If etag or last_modified is provided, the request will include the appropriate conditional headers. Returns the previous policy unchanged if 304 Not Modified is received.