robotxt 0.6.1

The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

/// Splits multiline comments into lines and prefixes them with `#`.
pub fn format_comment(txt: &str) -> String {
    txt.lines()
        .map(|txt| txt.trim())
        .filter(|txt| !txt.is_empty())
        .map(|txt| {
            if txt.starts_with('#') {
                txt.to_owned()
            } else {
                format!("# {txt}")
            }
        })
        .collect::<Vec<_>>()
        .join("\n")
}