Struct RobotFileParser 

pub struct RobotFileParser { /* private fields */ }

Reads, parses and answers questions about a single robots.txt file. Port of RobotFileParser.

Implementations

impl RobotFileParser

pub fn new(url: &str) -> Self

Create a parser for the robots.txt at url (pass "" for none). Port of __init__.

pub fn mtime(&self) -> bool

Whether the file has been parsed yet. Port of mtime (truthiness only — no real clock).

pub fn modified(&mut self)

Mark the file as just parsed. Port of modified.

pub fn set_url(&mut self, url: &str)

Set the robots.txt URL, caching its host and path. Port of set_url. A malformed authority leaves host/path empty rather than propagating (CPython would raise here).

pub fn url(&self) -> &str

The robots.txt URL set via new / set_url.

pub fn host(&self) -> &str

The host of the robots.txt URL.

pub fn path(&self) -> &str

The path of the robots.txt URL.

pub fn parse<S: AsRef<str>>(&mut self, lines: &[S])

Parse the lines of a robots.txt file (already split into lines, no trailing newlines). Port of parse.

pub fn can_fetch(&self, useragent: &str, url: &str) -> Result<bool, UrlError>

Decide whether useragent may fetch url, per the parsed robots.txt. Port of can_fetch.

Returns Err only when the URL’s authority is malformed (CPython raises ValueError from the internal urlsplit).

pub fn crawl_delay(&self, useragent: &str) -> Option<i64>

The Crawl-delay for useragent, or None. Port of crawl_delay.

pub fn request_rate(&self, useragent: &str) -> Option<RequestRate>

The Request-rate for useragent, or None. Port of request_rate.

pub fn site_maps(&self) -> Option<&[String]>

The Sitemap URLs, or None if there were none. Port of site_maps.

Trait Implementations

impl Clone for RobotFileParser

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RobotFileParser

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RobotFileParser

fn default() -> Self

Returns the “default value” for a type.

impl Display for RobotFileParser

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.