Module parser 

Structs

ParsedRobotsKey

A robots.txt has lines of key/value pairs. A ParsedRobotsKey represents a key.

RobotsTxtParser

A robotstxt parser.

Enums

ParseKeyType

A enum represents key types in robotstxt.

Functions

escape_pattern

Canonicalize the allowed/disallowed path patterns.