Expand description
Per-domain rate limiting to respect politeness constraints.
Token-bucket rate limiter with per-domain and global buckets, supporting crawl-delay from robots.txt and concurrency limiting.
Structs§
- Rate
Limiter - Token-bucket rate limiter for per-domain and global request throttling.
Enums§
- Rate
Limit Error - Errors that can occur during rate limiting.