Crate cylon[][src]



A compiler takes an input robots.txt file and outputs a compiled Cylon, which can be used to efficiently match a large number of paths against the robots.txt file.


A Cylon is a DFA that recognizes rules from a compiled robots.txt file. By providing it a URL path, it can decide whether or not the robots file that compiled it allows or disallows that path in roughly O(n) time, where n is the length of the path.