Struct HostDelay 

pub struct HostDelay { /* private fields */ }

The gap between starting two requests to the same host, see Crawler::with_host_delay: at least a minimum, growing towards a maximum while the host is throttled.

A host is throttled when it answers 429 Too Many Requests or 503 Service Unavailable, or a request to it times out. Once its concurrency cannot go lower, at a fixed limit or at one with HostConcurrency::adaptive, each round of throttling doubles its gap, from the minimum, or one second without one, up to the maximum. Each success halves it back to the minimum. A Retry-After from the host pauses it for exactly as long as it asks, beyond the maximum if need be, without widening its gap. The next request waits for the gap after the last one started, as the gap is when it is due, so a widened gap applies at once.

A Duration is the minimum, with the default maximum. A range sets both ends; the same value at both ends keeps the gap fixed.

use std::time::Duration;

use spire_core::crawler::Crawler;
use spire_core::backend::Noop;
use spire_core::worker::from_fn;

let worker = from_fn(|_: spire_core::Context<Noop>| async {});
let crawler = Crawler::new(Noop, worker)
    .with_host_delay(Duration::from_millis(200)..=Duration::from_secs(300));

Implementations

impl HostDelay

pub const DEFAULT_MAX: Duration

The maximum gap when only a minimum is given: one request a minute.

pub fn new(min: Duration, max: Duration) -> Self

A gap of at least min, growing up to max, or min if max is smaller.

pub const fn min(self) -> Duration

Returns the smallest gap.

pub const fn max(self) -> Duration

Returns the largest gap.

Trait Implementations

impl Clone for HostDelay

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for HostDelay

impl Debug for HostDelay

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for HostDelay

fn default() -> Self

No gap, growing up to DEFAULT_MAX while a host is throttled.

impl Eq for HostDelay

impl From<Duration> for HostDelay

fn from(min: Duration) -> Self

Converts to this type from the input type.

impl From<RangeInclusive<Duration>> for HostDelay

fn from(range: RangeInclusive<Duration>) -> Self

Converts to this type from the input type.

impl PartialEq for HostDelay

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for HostDelay