Module robots 

robots.txt policy enforcement. robots.txt honor for one-shot invocations (no cross-process cache).

Default policy is honor. Override requires BOTH --ignore-robots and --i-accept-robots-risk. Uses robotstxt::DefaultMatcher (Google port).

Enums

RobotsPolicy

Effective robots policy for one invocation.

Functions

enforce_robots

Fetch origin robots.txt and enforce honor policy (async, one-shot).

path_allowed

Returns true when path is allowed by a simple robots Disallow set. Empty disallow list means allow all. Exact prefix match on path.

scheme_skips_robots

Schemes that skip robots.txt (local / non-network).

url_allowed_by_robots_body

Check URL against robots.txt body for a user-agent using DefaultMatcher.