Struct AutoThrottle 

pub struct AutoThrottle { /* private fields */ }

Available on crate feature auto_throttle only.

Per-domain auto-throttle that dynamically computes crawl delay.

Thread-safe: DashMap for concurrent access, atomics for per-domain state.

Implementations

impl AutoThrottle

pub fn new(config: AutoThrottleConfig) -> Self

Create a new auto-throttle with the given configuration.

pub fn with_defaults() -> Self

Create with default configuration.

pub fn record_latency(&self, domain: &str, latency: Duration)

Record a response latency for a domain.

Call this after each successful (or failed) fetch with the elapsed wall-clock time. The EMA is updated atomically.

pub fn delay_for(&self, domain: &str) -> Duration

Compute the adaptive delay for a domain.

Formula: delay = ema_latency / target_concurrency, clamped to [min_delay_ms, max_delay_ms].

Returns Duration::ZERO if no samples have been recorded yet (cold-start: don’t delay until we have data).

pub fn latency_ms(&self, domain: &str) -> Option<f64>

Get the current EMA latency for a domain in milliseconds. Returns None if no samples recorded.

pub fn len(&self) -> usize

Number of tracked domains.

pub fn is_empty(&self) -> bool

Whether the throttle is tracking any domains.