Structsยง
- Compiler
- A compiler takes an input robots.txt file and outputs a compiled Cylon, which can be used to efficiently match a large number of paths against the robots.txt file.
- Cylon
- A Cylon is an NFA that recognizes rules from a compiled robots.txt file. By providing it a URL path, it can decide whether or not the robots file that compiled it allows or disallows that path.