Skip to main content

Module robots

Module robots 

Source
Expand description

robots.txt parser and per-domain cache (PRD §41).

Fetches, caches, and respects robots.txt for all domains. Cache entries expire after 24 hours (configurable).

Structs§

RobotRules
A parsed robots.txt directive set for a single user-agent.
RobotsCache
In-memory cache for robots.txt rules, keyed by domain.

Functions§

parse_robots_txt
Parse the text content of a robots.txt file for user-agent: *.