Struct RobotsRules 

pub struct RobotsRules { /* private fields */ }

Implementations

impl RobotsRules

pub fn parse(body: &[u8], agent: &str) -> RobotsRules

Parses a robots.txt body for agent. Lines it doesn’t understand are ignored.

pub fn from_status(status: u16) -> RobotsRules

Rules for a robots.txt that didn’t return 2xx. Like Google: a 4xx (except 429) means there are no rules; 429 and 5xx mean the whole site is off limits for now.

pub fn from_response(status: u16, body: &[u8], agent: &str) -> RobotsRules

pub fn allowed(&self, url_or_path: &str) -> bool

Takes an absolute URL or a path (with optional query).

pub fn blocks_everything(&self) -> bool

True when the homepage itself is off limits, so there is nothing to crawl.

pub fn crawl_delay(&self) -> Option<Duration>

Crawl-delay, capped at 10 seconds.

pub fn sitemaps(&self) -> &[String]