Module website_policy 

robots.txt-aware policy gate for web_fetch.

Before each outbound fetch, consult the host’s /robots.txt and honor Disallow entries for a sensible User-agent. Results cached per host.

Port of hermes-agent’s tools/website_policy.py, simplified:

• User-agent matching: we treat “MicroClaw” and “*” as equivalent
• Disallow + Allow rules; longest-prefix wins for conflicts
• Crawl-Delay read and surfaced via CrawlHint::delay
• Pages that 404 / 5xx / fail network → policy = allow (fail-open)

Structs

CrawlHint

Functions

consult_robots

Fetch (or reuse) robots.txt for the host of url and return a hint about whether the agent should proceed. Fail-open: network / 4xx / 5xx → allowed.