Struct RateLimiter 

pub struct RateLimiter { /* private fields */ }

Token-bucket rate limiter for per-domain and global request throttling.

Each domain gets its own token bucket. A global bucket throttles all requests across all domains. A request proceeds only when both the domain and global buckets have tokens. Supports crawl-delay from robots.txt and optional concurrency limiting.

Examples

use crawlkit_engine::ratelimit::RateLimiter;

let limiter = RateLimiter::new(2.0, 10.0);
assert!((limiter.per_domain_rps() - 2.0).abs() < f64::EPSILON);

Implementations

impl RateLimiter

pub fn new(per_domain_rps: f64, global_rps: f64) -> Self

Creates a new rate limiter with the given RPS settings.

pub fn from_crawl_config(config: &CrawlConfig) -> Self

Creates a rate limiter from a CrawlConfig.

pub fn with_concurrency(self, max_concurrent: usize) -> Self

Creates a rate limiter with a concurrency limit.

pub async fn acquire(&self, domain: &str) -> Result<(), RateLimitError>

Acquires permission to make a request to the given domain.

Blocks until tokens are available in both the domain and global buckets. Returns Err if the wait exceeds the timeout.

pub async fn acquire_with_timeout( &self, domain: &str, timeout: Duration, ) -> Result<(), RateLimitError>

Acquires permission with a timeout.

pub fn set_crawl_delay(&self, domain: &str, delay: Duration)

Sets the crawl-delay for a specific domain (from robots.txt).

This overrides the default RPS for that domain.

pub fn domain_tokens(&self, domain: &str) -> f64

Returns the current token count for a domain bucket. Creates the bucket if it doesn’t exist.

pub fn global_tokens(&self) -> f64

Returns the current global token count.

pub fn per_domain_rps(&self) -> f64

Returns the per-domain RPS setting.

pub fn global_rps(&self) -> f64

Returns the global RPS setting.