pub fn robots_fingerprint(body: &str) -> u64Expand description
Hash of the rules in a robots.txt body. Comments, blank lines and Sitemap: lines are
dropped and field names are lower-cased, so edits that don’t change what bots may do
don’t change the fingerprint. Values keep their case, since paths are case-sensitive.