parse_fetched_robots_txt

robotparser_fork::parser

Function parse_fetched_robots_txt 

Source 
pub fn parse_fetched_robots_txt(
    origin: Origin,
    status_code: u16,
    input: &str,
) -> ParseResult<FetchedRobotsTxt>
Expand description

Parses the text of the robots.txt file located in the specified place of origin, taking into account the response status code of the HTTP-request. IMPORTANT NOTE: origin must point to robots.txt url before redirects.