Struct RobotFileParser

pub struct RobotFileParser {
    pub disallow_all: bool,
    pub allow_all: bool,
    pub last_checked: i64,
    /* private fields */
}

robots.txt file parser

Fields

disallow_all: bool

Dis-allow links reguardless of robots.txt

allow_all: bool

Allow links reguardless of robots.txt

last_checked: i64

Time last checked robots.txt file

Implementations

impl RobotFileParser

pub fn new() -> Box<RobotFileParser>

Establish a new robotparser for a website domain

pub fn mtime(&self) -> i64

Returns the time the robots.txt file was last fetched.

This is useful for long-running web spiders that need to check for new robots.txt files periodically.

pub fn modified(&mut self)

Sets the time the robots.txt file was last fetched to the current time.

pub fn get_entries(&self) -> &Vec<Entry>

Get the entries inserted.

pub fn get_base_entry(&self) -> &Entry

Get the base entry inserted.

pub async fn read(&mut self, client: &Client, url: &str)

Reads the robots.txt URL and feeds it to the parser.

pub async fn from_response(&mut self, response: Response)

Reads the HTTP response and feeds it to the parser.

pub fn parse<T: AsRef<str>>(&mut self, lines: &[T])

Parse the input lines from a robots.txt file

We allow that a user-agent: line is not preceded by one or more blank lines.

pub fn set_disallow_list(&mut self, _path: &str)

Include the disallow paths in the regex set. This does nothing without the ‘regex’ feature.

pub fn set_disallow_agents_list(&mut self, _agent: &str)

Include the disallow agents in the regex set. This does nothing without the ‘regex’ feature.

pub fn build_disallow_list(&mut self)

Build the regex disallow list. This does nothing without the ‘regex’ feature.

pub fn can_fetch<T: AsRef<str>>(&self, useragent: T, url: &str) -> bool

Using the parsed robots.txt decide if useragent can fetch url

pub fn entry_allowed<T: AsRef<str>>(&self, useragent: &T, url_str: &str) -> bool

Is the entry apply to the robots.txt?

pub fn get_crawl_delay( &self, useragent: &Option<Box<CompactString>>, ) -> Option<Duration>

Returns the crawl delay for this user agent as a Duration, or None if no crawl delay is defined.

pub fn get_req_rate<T: AsRef<str>>(&self, useragent: T) -> Option<RequestRate>

Returns the request rate for this user agent as a RequestRate, or None if not request rate is defined

Trait Implementations

impl Clone for RobotFileParser

fn clone(&self) -> RobotFileParser

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RobotFileParser

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for RobotFileParser

fn eq(&self, other: &RobotFileParser) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for RobotFileParser

impl StructuralPartialEq for RobotFileParser