Struct RobotsTxt

pub struct RobotsTxt { /* private fields */ }

The robots.txt model that was obtained after parsing the text of the robots.txt file. To work with this model you should use the trait robotparser::service::RobotsTxtService. To create this structure you should use the robotparser::parser::parse_robots_txt.

Trait Implementations

impl Clone for RobotsTxt

fn clone(&self) -> RobotsTxt

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RobotsTxt

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl RobotsTxtService for RobotsTxt

fn can_fetch(&self, user_agent: &str, url: &Url) -> bool

Using the parsed robots.txt decide if useragent can fetch url.

fn get_crawl_delay(&self, user_agent: &str) -> Option<Duration>

Returns the crawl delay for this user agent as a Duration, or None if no crawl delay is defined.

fn normalize_url(&self, url: &mut Url) -> bool

Removes the request parameters from the url that were listed in the Clean-param directive. This method CHECKS that the origin of the transmitted url matches the origin of robots.txt. Returns true if the operation was applied to the passed url. In other cases it returns false.

fn normalize_url_ignore_origin(&self, url: &mut Url)

Removes the request parameters from the url that were listed in the Clean-param directive. This method DOES NOT CHECK that the origin of the transmitted url coincides with the origin of robots.txt.

fn get_sitemaps(&self) -> &[Url]

Returns the list of URL sitemaps that have been listed in the robots.txt file.

fn get_req_rate(&self, user_agent: &str) -> Option<RequestRate>

Returns information about the restrictions set for sending HTTP requests to the server.