Expand description
robots.txt parsing, caching, and compliance checking. robots.txt fetching, parsing, and caching.
Provides a RobotsTxtCache that fetches and parses robots.txt files
once per domain, then serves cached results to the crawl loop and
analyzers.
Structsยง
- Robots
TxtCache - Per-domain robots.txt cache.