Expand description
Main robots.txt checker with caching and fetching.
The RobotsChecker is the primary entry point for checking URLs against
robots.txt policies. It handles fetching, caching, parsing, and matching.
Structsยง
- Robots
Checker - Main robots.txt checker with caching and fetching.
- Robots
Diagnostics - Diagnostics for debugging robots.txt decisions.