Struct robotparser::RobotFileParser
[−]
[src]
pub struct RobotFileParser<'a> {
// some fields omitted
}robots.txt file parser
Methods
impl<'a> RobotFileParser<'a>[src]
fn new<T: AsRef<str>>(url: T) -> RobotFileParser<'a>
fn mtime(&self) -> i64
Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to check for new robots.txt files periodically.
fn modified(&self)
Sets the time the robots.txt file was last fetched to the current time.
fn set_url<T: AsRef<str>>(&mut self, url: T)
Sets the URL referring to a robots.txt file.
fn read(&self)
Reads the robots.txt URL and feeds it to the parser.
fn from_response(&self, response: &mut Response)
Reads the HTTP response and feeds it to the parser.
fn parse<T: AsRef<str>>(&self, lines: &[T])
Parse the input lines from a robots.txt file
We allow that a user-agent: line is not preceded by one or more blank lines.
fn can_fetch<T: AsRef<str>>(&self, useragent: T, url: T) -> bool
Using the parsed robots.txt decide if useragent can fetch url
fn get_crawl_delay<T: AsRef<str>>(&self, useragent: T) -> Option<Duration>
Returns the crawl delay for this user agent as a Duration, or None if no crawl delay is defined.
fn get_sitemaps<T: AsRef<str>>(&self, useragent: T) -> Vec<Url>
Returns the sitemaps for this user agent as a Vec<Url>.
fn get_req_rate<T: AsRef<str>>(&self, useragent: T) -> Option<RequestRate>
Returns the request rate for this user agent as a RequestRate, or None if not request rate is defined
Trait Implementations
impl<'a> Clone for RobotFileParser<'a>[src]
fn clone(&self) -> RobotFileParser<'a>
Returns a copy of the value. Read more
fn clone_from(&mut self, source: &Self)1.0.0
Performs copy-assignment from source. Read more
impl<'a> PartialEq for RobotFileParser<'a>[src]
fn eq(&self, __arg_0: &RobotFileParser<'a>) -> bool
This method tests for self and other values to be equal, and is used by ==. Read more
fn ne(&self, __arg_0: &RobotFileParser<'a>) -> bool
This method tests for !=.