Struct FetchedRobotsTxt

pub struct FetchedRobotsTxt { /* private fields */ }

A model of the robots.txt file that was downloaded over the network. This model takes into account HTTP response codes when loading the robots.txt file. To work with this model you should use the trait robotparser::service::RobotsTxtService. To create this structure you should use the robotparser::parser::parse_fetched_robots_txt.

Implementations

impl FetchedRobotsTxt

pub fn get_fetched_at(&self) -> &SystemTime

Returns the system time when the robots.txt file was downloaded over the network.

Trait Implementations

impl Clone for FetchedRobotsTxt

fn clone(&self) -> FetchedRobotsTxt

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FetchedRobotsTxt

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl RobotsTxtService for FetchedRobotsTxt

fn can_fetch(&self, user_agent: &str, url: &Url) -> bool

Using the parsed robots.txt decide if useragent can fetch url.

fn get_crawl_delay(&self, user_agent: &str) -> Option<Duration>

Returns the crawl delay for this user agent as a Duration, or None if no crawl delay is defined.

fn normalize_url(&self, url: &mut Url) -> bool

Removes the request parameters from the url that were listed in the Clean-param directive. This method CHECKS that the origin of the transmitted url matches the origin of robots.txt. Returns true if the operation was applied to the passed url. In other cases it returns false.

fn normalize_url_ignore_origin(&self, url: &mut Url)

Removes the request parameters from the url that were listed in the Clean-param directive. This method DOES NOT CHECK that the origin of the transmitted url coincides with the origin of robots.txt.

fn get_sitemaps(&self) -> &[Url]

Returns the list of URL sitemaps that have been listed in the robots.txt file.

fn get_req_rate(&self, user_agent: &str) -> Option<RequestRate>

Returns information about the restrictions set for sending HTTP requests to the server.