Module voyager::robots

source · [−]

Structs

Group

A Set of rules for a list of user-agents

RobotsData

Represents a robots.txt file

RobotsHandler

The handler that parses the robots.txt into a RobotsData

Rule

A rule that either allows or disallows an url pattern