Struct RobotsTxtParser 

pub struct RobotsTxtParser<'a, Handler: RobotsParseHandler> { /* private fields */ }

A robotstxt parser.

Implementations

impl<'a, Handler: RobotsParseHandler> RobotsTxtParser<'a, Handler>

pub fn new(robots_body: &'a str, handler: &'a mut Handler) -> Self

pub fn parse(&mut self)

Parse body of this Parser’s robots.txt and emit parse callbacks. This will accept typical typos found in robots.txt, such as ‘disalow’.

Note, this function will accept all kind of input but will skip everything that does not look like a robots directive.

pub fn parse_key_value(line: &str) -> (&str, &str, bool)

Attempts to parse a line of robots.txt into a key/value pair.

On success, the parsed key and value, and true, are returned. If parsing is unsuccessful, parse_key_value returns two empty strings and false.

pub fn need_escape_value_for_key(key: &ParsedRobotsKey) -> bool