Module matcher 

Structs

CachingRobotsMatcher

LongestMatchRobotsMatchStrategy

Implements the default robots.txt matching strategy. The maximum number of characters matched by a pattern is returned as its match priority.

Match

Instead of just maintaining a Boolean indicating whether a given line has matched, we maintain a count of the maximum number of characters matched by that pattern.

RobotsMatcher

RobotsMatcher - matches robots.txt against URLs.

Traits

RobotsMatchStrategy

Create a RobotsMatcher with the default matching strategy.