Function feroxbuster::extractor::extract_robots_txt

pub async fn extract_robots_txt(
    base_url: &str, 
    config: &Configuration, 
    tx_stats: UnboundedSender<StatCommand>
) -> HashSet<String>

Entry point to perform link extraction from robots.txt

base_url can have paths and subpaths, however robots.txt will be requested from the root of the url given the url: http://localhost/stuff/things this function requests: http://localhost/robots.txt