pub struct FetchConfig {Show 17 fields
pub concurrency: usize,
pub timeout: Duration,
pub connect_timeout: Duration,
pub max_redirects: usize,
pub max_body_bytes: usize,
pub retries: u32,
pub retry_backoff: Duration,
pub per_host_delay: Duration,
pub respect_robots: bool,
pub respect_crawl_delay: bool,
pub impersonate: Impersonate,
pub accept_language: String,
pub browser_fallback: bool,
pub proxies: Vec<String>,
pub max_retry_after: Duration,
pub cookie_file: Option<PathBuf>,
pub allow_private_hosts: bool,
}Expand description
Everything tunable about fetching.
Fields§
§concurrency: usizeMaximum requests in flight across all hosts.
timeout: DurationTotal per-request deadline.
connect_timeout: DurationConnection establishment deadline.
max_redirects: usizeRedirects followed before giving up.
max_body_bytes: usizeHard cap on a response body. Enforced while streaming, so an oversized body costs one chunk, not the whole download.
retries: u32Retries for retryable failures.
retry_backoff: DurationBase delay for exponential backoff between retries.
per_host_delay: DurationMinimum spacing between two requests to the same host.
respect_robots: boolHonour robots.txt. Leave this on unless you own the target.
respect_crawl_delay: boolHonour a Crawl-delay larger than FetchConfig::per_host_delay.
impersonate: ImpersonateBrowser fingerprint to present.
accept_language: StringAccept-Language header.
browser_fallback: boolEscalate JS-gated pages to headless Chrome. Needs the browser feature.
proxies: Vec<String>Proxies to route requests through, rotated round-robin.
The only real answer to IP-reputation blocking: a TLS fingerprint says
what your client is, and an address says who it is. Accepts http://,
https:// and socks5:// URLs, with optional user:pass@.
max_retry_after: DurationHonour a Retry-After header, up to this long. Set to zero to ignore it.
File to persist cookies to, so a session survives the process.
allow_private_hosts: boolAllow loopback, private and link-local addresses as fetch targets.
Off by default. A search result or a redirect can point at
169.254.169.254 or at a service bound to the machine running this,
and whatever comes back is extracted into a context window destined for
a model – which turns an ordinary crawl into a way to read the cloud
metadata endpoint out loud. Turn it on to crawl your own development
server.
Trait Implementations§
Source§impl Clone for FetchConfig
impl Clone for FetchConfig
Source§impl Debug for FetchConfig
Hand-written rather than derived: FetchConfig::proxies can carry
user:pass@, and a derived Debug would put those credentials into any log
that dumps a Fetcher or a Pipeline. Shows the fields worth seeing in a
dump; .. covers the rest, so a new field cannot reintroduce a leak.
impl Debug for FetchConfig
Hand-written rather than derived: FetchConfig::proxies can carry
user:pass@, and a derived Debug would put those credentials into any log
that dumps a Fetcher or a Pipeline. Shows the fields worth seeing in a
dump; .. covers the rest, so a new field cannot reintroduce a leak.
Auto Trait Implementations§
impl Forget for FetchConfig
impl Freeze for FetchConfig
impl Move for FetchConfig
impl RefUnwindSafe for FetchConfig
impl Send for FetchConfig
impl Sync for FetchConfig
impl Unpin for FetchConfig
impl UnsafeUnpin for FetchConfig
impl UnwindSafe for FetchConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more