Module robotstxt_with_cache::parser [−][src]
Structs
ParsedRobotsKey | A robots.txt has lines of key/value pairs. A ParsedRobotsKey represents a key. |
RobotsTxtParser | A robotstxt parser. |
Enums
ParseKeyType | A enum represents key types in robotstxt. |
Functions
escape_pattern | Canonicalize the allowed/disallowed path patterns. |