Function texting_robots::get_robots_url

pub fn get_robots_url(url: &str) -> Result<String, ParseError>

Construct the URL for robots.txt when given a base URL from the target domain.

Errors

If there are any issues in parsing the URL, a ParseError from the URL crate will be returned.

use texting_robots::get_robots_url;

let robots_url = get_robots_url("https://example.com/abc/file.html").unwrap();
assert_eq!(robots_url, "https://example.com/robots.txt");