Expand description
A small, strict robots.txt reader.
Being a good citizen is not optional for a library people will point at
other people’s servers, so this is on by default. The implementation follows
the parts of RFC 9309 that matter in practice: user-agent grouping, longest
matching rule wins, Allow beats Disallow on equal length, * and $
wildcards, and Crawl-delay as a courtesy extension.
Structs§
- Robots
- Parsed rules for one user-agent.