pub fn path_allowed(path: &str, disallows: &[&str]) -> bool
Returns true when path is allowed by a simple robots Disallow set. Empty disallow list means allow all. Exact prefix match on path.