Function robotstxt::parse_robotstxt

pub fn parse_robotstxt(
    robots_body: &str, 
    parse_callback: &mut impl RobotsParseHandler
)

Parses body of a robots.txt and emits parse callbacks. This will accept typical typos found in robots.txt, such as 'disalow'.

Note, this function will accept all kind of input but will skip everything that does not look like a robots directive.