Module parser

Module parser 

Source

Structs§

ParsedRobotsKey
A robots.txt has lines of key/value pairs. A ParsedRobotsKey represents a key.
RobotsTxtParser
A robotstxt parser.

Enums§

ParseKeyType
A enum represents key types in robotstxt.

Functions§

escape_pattern
Canonicalize the allowed/disallowed path patterns.