pub struct Fetcher { /* private fields */ }Expand description
A pooled, polite, fingerprint-aware HTTP client.
Implementations§
Source§impl Fetcher
impl Fetcher
Sourcepub fn with_config(cfg: FetchConfig) -> Result<Self>
pub fn with_config(cfg: FetchConfig) -> Result<Self>
Build a fetcher from an explicit config.
Write the cookie jar to FetchConfig::cookie_file.
Clearance cookies are the expensive part of getting through a bot wall; throwing them away when the process exits means paying for them again.
Sourcepub fn config(&self) -> &FetchConfig
pub fn config(&self) -> &FetchConfig
The configuration this fetcher was built with.
Sourcepub async fn fetch(&self, url: &str) -> Result<Page>
pub async fn fetch(&self, url: &str) -> Result<Page>
Fetch one URL, honouring robots.txt when configured to.
Sourcepub async fn fetch_api(&self, url: &str) -> Result<Page>
pub async fn fetch_api(&self, url: &str) -> Result<Page>
Fetch a provider endpoint, skipping the robots.txt check.
robots.txt governs crawling — the discovery of documents a site did
not hand you. A search or feed endpoint the caller named explicitly is
a query, not a crawl, and several of them (DuckDuckGo’s HTML endpoint
among them) disallow the very path they exist to serve. Pages found
through those endpoints go through Fetcher::fetch and are checked
normally, which is where the rule actually protects anyone.
Trait Implementations§
Auto Trait Implementations§
impl !RefUnwindSafe for Fetcher
impl !UnwindSafe for Fetcher
impl Forget for Fetcher
impl Freeze for Fetcher
impl Move for Fetcher
impl Send for Fetcher
impl Sync for Fetcher
impl Unpin for Fetcher
impl UnsafeUnpin for Fetcher
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more