pub struct FetchConfig {Show 16 fields
pub concurrency: usize,
pub timeout: Duration,
pub connect_timeout: Duration,
pub max_redirects: usize,
pub max_body_bytes: usize,
pub retries: u32,
pub retry_backoff: Duration,
pub per_host_delay: Duration,
pub respect_robots: bool,
pub respect_crawl_delay: bool,
pub impersonate: Impersonate,
pub accept_language: String,
pub browser_fallback: bool,
pub proxies: Vec<String>,
pub max_retry_after: Duration,
pub cookie_file: Option<PathBuf>,
}Expand description
Everything tunable about fetching.
Fields§
§concurrency: usizeMaximum requests in flight across all hosts.
timeout: DurationTotal per-request deadline.
connect_timeout: DurationConnection establishment deadline.
max_redirects: usizeRedirects followed before giving up.
max_body_bytes: usizeHard cap on a response body. Enforced while streaming, so an oversized body costs one chunk, not the whole download.
retries: u32Retries for retryable failures.
retry_backoff: DurationBase delay for exponential backoff between retries.
per_host_delay: DurationMinimum spacing between two requests to the same host.
respect_robots: boolHonour robots.txt. Leave this on unless you own the target.
respect_crawl_delay: boolHonour a Crawl-delay larger than FetchConfig::per_host_delay.
impersonate: ImpersonateBrowser fingerprint to present.
accept_language: StringAccept-Language header.
browser_fallback: boolEscalate JS-gated pages to headless Chrome. Needs the browser feature.
proxies: Vec<String>Proxies to route requests through, rotated round-robin.
The only real answer to IP-reputation blocking: a TLS fingerprint says
what your client is, and an address says who it is. Accepts http://,
https:// and socks5:// URLs, with optional user:pass@.
max_retry_after: DurationHonour a Retry-After header, up to this long. Set to zero to ignore it.
File to persist cookies to, so a session survives the process.
Trait Implementations§
Source§impl Clone for FetchConfig
impl Clone for FetchConfig
Source§impl Debug for FetchConfig
impl Debug for FetchConfig
Auto Trait Implementations§
impl Freeze for FetchConfig
impl RefUnwindSafe for FetchConfig
impl Send for FetchConfig
impl Sync for FetchConfig
impl Unpin for FetchConfig
impl UnsafeUnpin for FetchConfig
impl UnwindSafe for FetchConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more