Expand description
Types - Core types for robots.txt handling (RFC 9309)
Structs§
- Compiled
Pattern - Compiled pattern for efficient matching
- Decision
- Decision returned by is_allowed
- Effective
Rules - Effective rules for a specific user-agent (after merging groups)
- Group
- A group of rules for specific user-agents
- Request
Rate - Request-rate directive (non-standard but common) Format: “requests/seconds” e.g., “1/10” means 1 request per 10 seconds
- Robots
Cache Key - Cache key for robots.txt (scheme + authority)
- Robots
Config - Configuration for robots.txt handling
- Robots
Policy - Complete robots.txt policy
- Rule
- A single rule (Allow or Disallow)
Enums§
- Decision
Reason - Reason for the decision
- Fetch
Status - Fetch status distinguishing Unavailable vs Unreachable (RFC 9309)
- Rule
Kind - Rule kind