Function feroxbuster::extractor::request_robots_txt

pub async fn request_robots_txt(
    base_url: &str, 
    config: &Configuration, 
    tx_stats: UnboundedSender<StatCommand>
) -> Option<FeroxResponse>

helper function that simply requests /robots.txt on the given url's base url

example: http://localhost/api/users -> http://localhost/robots.txt

The length of the given path has no effect on what's requested; it's always base url + /robots.txt