Function fetch_robots 

pub async fn fetch_robots(
    fetcher: &Fetcher,
    url: &Url,
) -> Result<(RobotsRules, RobotsFile), FetchError>

Fetches /robots.txt for the site url belongs to. Connection-level failures are returned as errors; HTTP error statuses become rules.