Function robotstxt_with_cache::parse_robotstxt[][src]

pub fn parse_robotstxt(
    robots_body: &str,
    parse_callback: &mut impl RobotsParseHandler
)
Expand description

Parses body of a robots.txt and emits parse callbacks. This will accept typical typos found in robots.txt, such as ‘disalow’.

Note, this function will accept all kind of input but will skip everything that does not look like a robots directive.