Module matcher 

Matcher - RFC 9309 compliant path matching

Structs

RobotsMatcher

Matcher for checking if paths are allowed

Functions

get_crawl_delay

Get crawl delay for a user-agent (convenience function)

is_allowed

Check if a URL is allowed (convenience function)