Struct FetcherConfig 

pub struct FetcherConfig {

    pub timeout_secs: u64,
    pub retries: u32,
    pub retry_delay_secs: u64,
    pub follow_redirects: bool,
    pub max_redirects: u32,
    pub verify_ssl: bool,
    pub proxy: Option<String>,
    pub proxies: HashMap<String, String>,
    pub proxy_list: Vec<String>,
    pub headers: HashMap<String, String>,
    pub stealthy_headers: bool,
    pub user_agent: Option<String>,
    pub max_body_bytes: usize,
}

Configuration for HTTP fetchers.

Build one with FetcherConfig::builder or start from FetcherConfig::default (30s timeout, 3 retries with a 1s delay, redirects followed up to 10 hops, SSL verified, stealth headers on, 50 MiB body cap, no proxies).

Fields

timeout_secs: u64

Total per-request timeout in seconds (connect through body read).

retries: u32

How many times a failed request is retried (in addition to the initial attempt).

retry_delay_secs: u64

Fixed delay in seconds between retry attempts.

follow_redirects: bool

Whether HTTP redirects are followed at all.

max_redirects: u32

Maximum number of redirect hops when follow_redirects is on.

verify_ssl: bool

Whether TLS certificates are verified. Disabling this is insecure and logs a warning when the fetcher is built.

proxy: Option<String>

A single proxy URL applied to all protocols, used as a fallback after the per-protocol proxies map.

proxies: HashMap<String, String>

Per-protocol proxy map: keys "http" / "https" bind a proxy to that scheme, any other key acts as a wildcard for all protocols.

proxy_list: Vec<String>

Proxy URLs to rotate through, one HTTP client is built per entry and selected round-robin per request. Takes precedence over proxy / proxies when non-empty.

headers: HashMap<String, String>

Extra request headers (lowercase names). These win over the generated stealth headers.

stealthy_headers: bool

Whether browser-like stealth headers are generated for each request (see FetcherConfig::build_headers).

user_agent: Option<String>

Fixed User-Agent. When None, a user agent is picked deterministically per URL from a small pool of real browser strings.

max_body_bytes: usize

Maximum response body size in bytes. Responses larger than this are aborted to protect against OOM. Defaults to 50 MiB.

Implementations

impl FetcherConfig

pub fn builder() -> FetcherConfigBuilder

Returns a builder initialised with default values.

pub fn build_headers(&self, url: &str, stealth: bool) -> HashMap<String, String>

Build the HTTP headers to send for a request.

When stealth is true the function adds several browser-like headers that make the request harder to fingerprint as a bot.

Trait Implementations

impl Clone for FetcherConfig

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FetcherConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for FetcherConfig

fn default() -> Self

Returns the “default value” for a type.