Module robots 

robots.txt rules for CodoSEObot, following Google’s handling of groups, status codes and size.

Matching is linear: rules are * wildcards with an optional trailing $, matched greedily against the path and query with plain substring search. A hostile file can’t cost more than MAX_RULES rules of at most MAX_PATTERN_LEN bytes each; anything beyond that is skipped, not the whole file.

Structs

RobotsRules

Constants

MAX_PATTERN_LEN

MAX_RULES

ROBOTS_AGENT

The product token sites use to address us in robots.txt.

Functions

fetch_robots

Fetches /robots.txt for the site url belongs to. Connection-level failures are returned as errors; HTTP error statuses become rules.