Expand description
Parser - RFC 9309 compliant robots.txt parser
This module provides a robust parser for robots.txt files that handles:
- UTF-8 BOM stripping
- Request-rate directive (non-standard but common)
- Proper percent-encoding normalization
- Size limits per RFC 9309
Modules§
- encoding
- Utility functions for percent-encoding handling
Structs§
- Robots
Parser - Parser for robots.txt files
Constants§
- MAX_
ROBOTS_ SIZE - Maximum robots.txt size (RFC 9309 requires at least 500 KiB)