Struct RobotsMatcher

pub struct RobotsMatcher<S: RobotsMatchStrategy> { /* private fields */ }

RobotsMatcher - matches robots.txt against URLs.

The Matcher uses a default match strategy for Allow/Disallow patterns which is the official way of Google crawler to match robots.txt. It is also possible to provide a custom match strategy.

The entry point for the user is to call one of the allowed_by_robots methods that return directly if a URL is being allowed according to the robots.txt and the crawl agent. The RobotsMatcher can be re-used for URLs/robots.txt but is not thread-safe.

Implementations

impl<'a, S: RobotsMatchStrategy> RobotsMatcher<S>

pub fn allowed_by_robots( &mut self, robots_body: &str, user_agents: Vec<&str>, url: &str, ) -> bool

where Self: RobotsParseHandler,

Returns true if ‘url’ is allowed to be fetched by any member of the “user_agents” vector. ‘url’ must be %-encoded according to RFC3986.

pub fn one_agent_allowed_by_robots( &mut self, robots_txt: &str, user_agent: &str, url: &str, ) -> bool

where Self: RobotsParseHandler,

Do robots check for ‘url’ when there is only one user agent. ‘url’ must be %-encoded according to RFC3986.

pub fn is_valid_user_agent_to_obey(user_agent: &str) -> bool

Verifies that the given user agent is valid to be matched against robots.txt. Valid user agent strings only contain the characters [a-zA-Z_-].

pub fn matching_line(&self) -> u32

Returns the line that matched or 0 if none matched.

Trait Implementations

impl<S: Default + RobotsMatchStrategy> Default for RobotsMatcher<S>

fn default() -> RobotsMatcher<S>

Returns the “default value” for a type.

impl<S: RobotsMatchStrategy> RobotsParseHandler for RobotsMatcher<S>

fn handle_robots_start(&mut self)

fn handle_robots_end(&mut self)

fn handle_user_agent(&mut self, _line_num: u32, user_agent: &str)

fn handle_allow(&mut self, line_num: u32, value: &str)

fn handle_disallow(&mut self, line_num: u32, value: &str)

fn handle_sitemap(&mut self, _line_num: u32, _value: &str)

fn handle_unknown_action(&mut self, _line_num: u32, _action: &str, _value: &str)

Any other unrecognized name/value pairs.