Crate cylon

Crate cylon 

Source

Structsยง

Compiler
A compiler takes an input robots.txt file and outputs a compiled Cylon, which can be used to efficiently match a large number of paths against the robots.txt file.
Cylon
A Cylon is an NFA that recognizes rules from a compiled robots.txt file. By providing it a URL path, it can decide whether or not the robots file that compiled it allows or disallows that path.