Struct Limiter 

pub struct Limiter { /* private fields */ }

Paces the requests to one site and shares a global in-flight cap with other sites.

Implementations

impl Limiter

pub fn new( p: &Politeness, crawl_delay: Option<Duration>, global: Arc<Semaphore>, ) -> Limiter

The gap between requests is the slower of 1 / requests_per_sec and the site’s Crawl-delay, never above Politeness::max_crawl_delay.

pub async fn acquire(&self) -> Permit

Waits for a site connection, then for this request’s start time, then for a global slot. The start time is reserved before waiting, so concurrent callers get evenly spaced slots. A pause set while waiting (a 429 or 503 on another connection) still applies: the request then reserves a new slot after it.

pub fn on_response(&self, status: u16, retry_after: Option<Duration>)

Reports a response. A 429 or 503 doubles the interval (up to the cap) and holds back the next requests for retry_after, or two new intervals when the site gave none.

pub fn set_crawl_delay(&self, delay: Option<Duration>)

Applies a Crawl-delay learned after creation (robots.txt is read through this limiter). The gap becomes the slower of 1 / requests_per_sec and delay, capped as in Limiter::new; a longer gap from a 429/503 back-off is kept.

pub fn interval(&self) -> Duration

The current gap between requests.