Module robotstxt::parser [−][src]
A parser module.
Structs
| ParsedRobotsKey | A robots.txt has lines of key/value pairs. A ParsedRobotsKey represents a key. |
| RobotsTxtParser | A robotstxt parser. |
Enums
| ParseKeyType | A enum represents key types in robotstxt. |
Functions
| escape_pattern | Canonicalize the allowed/disallowed path patterns. |