Struct Identities 

pub struct Identities { /* private fields */ }

The identities of a crawl, and how they are picked, set aside and replaced.

A request pinned with Seed::with_identity goes out as that identity; any other gets one picked in turn, at random, or per host.

An identity whose requests fail max_failures times in a row, through connection failures, timeouts, or 403, 407 or 429 answers, is taken out of use: a generated one is retired and replaced by the next, a listed one set aside for a while, longer each time.

Implementations

impl Identities

pub const DEFAULT_MAX_ACTIVE: usize = 16

The identities generated at once by default, see max_active.

pub fn new(identities: impl IntoIterator<Item = Identity>) -> Self

Uses a fixed list of identities.

Panics

Panics if identities is empty or two share a name.

pub fn from_fn<F>(generate: F) -> Self

where F: Fn(u64) -> Result<Identity> + Send + Sync + 'static,

Builds identities on demand: identity n, counting from 0, from generate, e.g. a gateway’s session n, or the nth proxy of a list. Up to max_active are in use at once; one taken out of use is replaced by the next n.

Names should follow from n, so a resumed crawl builds the same identities again. A failure to build one fails the request that needed it.

pub fn max_active(self, n: usize) -> Self

Generates at most n identities at once, created as requests need them, so a thousand cost nothing until used. Ignored for a fixed list.

Panics

Panics if n is zero.

pub const fn in_turn(self) -> Self

Picks identities in turn, one request each. This is the default.

pub const fn at_random(self) -> Self

Picks an identity at random for every request.

pub const fn sticky_per_host(self) -> Self

Keeps each host on one identity, as one visitor would be, for as long as that identity stays in use; a new host gets a new identity while fewer than max_active are.

pub const fn retire_after_requests(self, n: u64) -> Self

Retires a generated identity after n requests, replacing it with the next.

pub const fn retire_after(self, age: Duration) -> Self

Retires a generated identity once it is age old, replacing it with the next.

pub fn max_failures(self, n: u32) -> Self

Takes an identity out of use after n failures in a row, 3 by default.

Panics

Panics if n is zero.

pub const fn rest(self, duration: Duration) -> Self

Sets a failing listed identity aside for duration the first time, one minute by default, and twice as long each time after.

Trait Implementations

impl Clone for Identities

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Identities

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.