Struct RobotFileParser 

pub struct RobotFileParser {
    pub disallow_all: bool,
    pub allow_all: bool,
    pub last_checked: i64,
    pub disallow_paths_regex: RegexSet,
    pub disallow_paths: HashSet<String>,
    pub disallow_agents_regex: RegexSet,
    pub wild_card_agent: bool,
    pub disallow_agents: HashSet<String>,
    /* private fields */
}

robots.txt file parser

Fields

disallow_all: bool

Dis-allow links reguardless of robots.txt

allow_all: bool

Allow links reguardless of robots.txt

last_checked: i64

Time last checked robots.txt file

disallow_paths_regex: RegexSet

Available on crate feature regex only.

Disallow list of regex paths to ignore.

disallow_paths: HashSet<String>

Available on crate feature regex only.

Disallow list of paths to ignore.

disallow_agents_regex: RegexSet

Available on crate feature regex only.

Disallow list of regex agents to ignore.

wild_card_agent: bool

Available on crate feature regex only.

Wild card agent provided.

disallow_agents: HashSet<String>

Available on crate feature regex only.

Disallow list of agents to ignore.

Implementations

impl RobotFileParser

pub fn new() -> Box<RobotFileParser>

Available on crate feature regex only.

Establish a new robotparser for a website domain

pub fn mtime(&self) -> i64

Returns the time the robots.txt file was last fetched.

This is useful for long-running web spiders that need to check for new robots.txt files periodically.

pub fn modified(&mut self)

Sets the time the robots.txt file was last fetched to the current time.

pub fn get_entries(&self) -> &Vec<Entry>

Get the entries inserted.

pub fn get_base_entry(&self) -> &Entry

Get the base entry inserted.

pub async fn read(&mut self, client: &Client, url: &str)

Reads the robots.txt URL and feeds it to the parser.

pub async fn from_response(&mut self, response: Response)

Reads the HTTP response and feeds it to the parser.

pub fn parse_str(&mut self, text: &str)

Parse a robots.txt string directly, splitting lines via memchr to avoid allocating an intermediate Vec<&str>.

pub fn parse<T: AsRef<str>>(&mut self, lines: &[T])

Parse the input lines from a robots.txt file

We allow that a user-agent: line is not preceded by one or more blank lines.

pub fn set_disallow_list(&mut self, path: &str)

Available on crate feature regex only.

Include the disallow paths in the regex set. This does nothing without the ‘regex’ feature.

pub fn set_disallow_agents_list(&mut self, agent: &str)

Available on crate feature regex only.

Include the disallow agents in the regex set. This does nothing without the ‘regex’ feature.

pub fn build_disallow_list(&mut self)

Available on crate feature regex only.

Build the regex disallow list. This does nothing without the ‘regex’ feature.

pub fn can_fetch<T: AsRef<str>>(&self, useragent: T, url: &str) -> bool

Using the parsed robots.txt decide if useragent can fetch url

pub fn entry_allowed<T: AsRef<str>>(&self, useragent: &T, url_str: &str) -> bool

Available on crate feature regex only.

Is the entry apply to the robots.txt?

pub fn get_crawl_delay( &self, useragent: &Option<Box<CompactString>>, ) -> Option<Duration>

Returns the crawl delay for this user agent as a Duration, or None if no crawl delay is defined.

pub fn get_req_rate<T: AsRef<str>>(&self, useragent: T) -> Option<RequestRate>

Returns the request rate for this user agent as a RequestRate, or None if not request rate is defined

Trait Implementations

impl Clone for RobotFileParser

fn clone(&self) -> RobotFileParser

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RobotFileParser

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.