Structs
A compiler takes an input robots.txt file and outputs a compiled Cylon, which can be used to efficiently match a large number of paths against the robots.txt file.
A Cylon is an NFA that recognizes rules from a compiled robots.txt file. By providing it a URL path, it can decide whether or not the robots file that compiled it allows or disallows that path.