Skip to main content

Module robots

Module robots 

Source
Expand description

A small, strict robots.txt reader.

Being a good citizen is not optional for a library people will point at other people’s servers, so this is on by default. The implementation follows the parts of RFC 9309 that matter in practice: user-agent grouping, longest matching rule wins, Allow beats Disallow on equal length, * and $ wildcards, and Crawl-delay as a courtesy extension.

Structs§

Robots
Parsed rules for one user-agent.