Module robots 

robots.txt policy enforcement. robots.txt honor for one-shot invocations (no cross-process cache).

Default policy is honor. Override requires BOTH --ignore-robots and --i-accept-robots-risk. Uses robotstxt::DefaultMatcher (Google port).

Workload

I/O-bound (network GET of /robots.txt). Reuses process-wide [shared_http_client] so keep-alive / TLS session cache survive batch scrapes. Not CPU-bound → no rayon. Not a long-lived daemon → no connection pool beyond reqwest’s internal pool for this one-shot process.

Enums

RobotsPolicy

Effective robots policy for one invocation.

Functions

enforce_robots

Fetch origin robots.txt and enforce honor policy (async, one-shot).

path_allowed

Returns true when path is allowed by a simple robots Disallow set. Empty disallow list means allow all. Exact prefix match on path.

scheme_skips_robots

Schemes that skip robots.txt (local / non-network).

shared_http_client

Process-wide async HTTP client (rules: create reqwest::Client once, reuse).

url_allowed_by_robots_body

Check URL against robots.txt body for a user-agent using DefaultMatcher.