Struct robotparser::RobotFileParser

pub struct RobotFileParser<'a> { /* fields omitted */ }

robots.txt file parser

Methods

impl<'a> RobotFileParser<'a>

fn new<T: AsRef<str>>(url: T) -> RobotFileParser<'a>

fn mtime(&self) -> i64

Returns the time the robots.txt file was last fetched.

This is useful for long-running web spiders that need to check for new robots.txt files periodically.

fn modified(&self)

Sets the time the robots.txt file was last fetched to the current time.

fn set_url<T: AsRef<str>>(&mut self, url: T)

Sets the URL referring to a robots.txt file.

fn read(&self)

Reads the robots.txt URL and feeds it to the parser.

fn from_response(&self, response: &mut Response)

Reads the HTTP response and feeds it to the parser.

fn parse<T: AsRef<str>>(&self, lines: &[T])

Parse the input lines from a robots.txt file

We allow that a user-agent: line is not preceded by one or more blank lines.

fn can_fetch<T: AsRef<str>>(&self, useragent: T, url: T) -> bool

Using the parsed robots.txt decide if useragent can fetch url

fn get_crawl_delay<T: AsRef<str>>(&self, useragent: T) -> Option<Duration>

Returns the crawl delay for this user agent as a Duration, or None if no crawl delay is defined.

fn get_sitemaps<T: AsRef<str>>(&self, useragent: T) -> Vec<Url>

Returns the sitemaps for this user agent as a Vec<Url>.

fn get_req_rate<T: AsRef<str>>(&self, useragent: T) -> Option<RequestRate>

Returns the request rate for this user agent as a RequestRate, or None if not request rate is defined

Trait Implementations

impl<'a> Debug for RobotFileParser<'a>

fn fmt(&self, __arg_0: &mut Formatter) -> Result

Formats the value using the given formatter.

impl<'a> Eq for RobotFileParser<'a>

impl<'a> PartialEq for RobotFileParser<'a>

fn eq(&self, __arg_0: &RobotFileParser<'a>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, __arg_0: &RobotFileParser<'a>) -> bool

This method tests for !=.

impl<'a> Clone for RobotFileParser<'a>

fn clone(&self) -> RobotFileParser<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.