Struct robotstxt::matcher::RobotsMatcher

pub struct RobotsMatcher<'a, S: RobotsMatchStrategy> { /* fields omitted */ }

RobotsMatcher - matches robots.txt against URLs.

The Matcher uses a default match strategy for Allow/Disallow patterns which is the official way of Google crawler to match robots.txt. It is also possible to provide a custom match strategy.

The entry point for the user is to call one of the allowed_by_robots methods that return directly if a URL is being allowed according to the robots.txt and the crawl agent. The RobotsMatcher can be re-used for URLs/robots.txt but is not thread-safe.

Implementations

impl<'a, S: RobotsMatchStrategy> RobotsMatcher<'a, S>

pub fn allowed_by_robots(
    &mut self,
    robots_body: &'a str,
    user_agents: Vec<&'a str>,
    url: &'a str
) -> bool where
    Self: RobotsParseHandler, 

Returns true if 'url' is allowed to be fetched by any member of the "user_agents" vector. 'url' must be %-encoded according to RFC3986.

pub fn one_agent_allowed_by_robots(
    &mut self,
    robots_txt: &'a str,
    user_agent: &'a str,
    url: &'a str
) -> bool where
    Self: RobotsParseHandler, 

Do robots check for 'url' when there is only one user agent. 'url' must be %-encoded according to RFC3986.

pub fn is_valid_user_agent_to_obey(user_agent: &str) -> bool

Verifies that the given user agent is valid to be matched against robots.txt. Valid user agent strings only contain the characters [a-zA-Z_-].

Trait Implementations

impl<'a, S: Default + RobotsMatchStrategy> Default for RobotsMatcher<'a, S>

fn default() -> RobotsMatcher<'a, S>

Returns the "default value" for a type.

impl<S: RobotsMatchStrategy> RobotsParseHandler for RobotsMatcher<'_, S>

fn handle_robots_start(&mut self)

fn handle_robots_end(&mut self)

fn handle_user_agent(&mut self, line_num: u32, user_agent: &str)

fn handle_allow(&mut self, line_num: u32, value: &str)

fn handle_disallow(&mut self, line_num: u32, value: &str)

fn handle_sitemap(&mut self, line_num: u32, value: &str)

fn handle_unknown_action(&mut self, line_num: u32, action: &str, value: &str)

Any other unrecognized name/value pairs.