Expand description
robots.txt parser and per-domain cache (PRD §41).
Fetches, caches, and respects robots.txt for all domains. Cache entries expire after 24 hours (configurable).
Structs§
- Robot
Rules - A parsed robots.txt directive set for a single user-agent.
- Robots
Cache - In-memory cache for robots.txt rules, keyed by domain.
Functions§
- parse_
robots_ txt - Parse the text content of a robots.txt file for
user-agent: *.