A Set of rules for a list of user-agents
Represents a robots.txt file
robots.txt
The handler that parses the robots.txt into a RobotsData
RobotsData
A rule that either allows or disallows an url pattern