pub struct HttpClient { /* private fields */ }Expand description
Shared HTTP client with connection pooling, retries, and rate limiting.
Implementations§
Source§impl HttpClient
impl HttpClient
Sourcepub fn new(config: &FetchiumConfig) -> FetchiumResult<Self>
pub fn new(config: &FetchiumConfig) -> FetchiumResult<Self>
Create a new HTTP client from config.
Sourcepub fn client_for_domain_with_locale(
&self,
domain: &str,
locale: Option<&str>,
) -> Client
pub fn client_for_domain_with_locale( &self, domain: &str, locale: Option<&str>, ) -> Client
Get a reqwest client for a specific domain.
Priority order:
- DataImpulse residential proxy (country-targeted) — Google only
- Direct connection — default path for everything else
Cheap path by default: non-Google traffic stays direct until a backend explicitly asks for proxy escalation.
Sourcepub fn client_for_domain(&self, domain: &str) -> Client
pub fn client_for_domain(&self, domain: &str) -> Client
Backwards-compatible wrapper — no locale hint.
Sourcepub fn client_direct(&self) -> Client
pub fn client_direct(&self) -> Client
Direct client — bypasses ALL proxy routing (residential and datacenter).
Use for first-attempt requests where proxy is not yet needed.
If the direct attempt fails, escalate to client_for_domain_with_locale.
Saves DataImpulse GB when direct connections succeed (majority of requests).
Sourcepub fn proxy_client_for_domain_with_locale(
&self,
domain: &str,
locale: Option<&str>,
) -> Client
pub fn proxy_client_for_domain_with_locale( &self, domain: &str, locale: Option<&str>, ) -> Client
Get a forced proxy client for a domain.
Google uses residential. All other proxy-eligible domains use the fixed proxy pool. Falls back to direct when no proxy is available.
Sourcepub fn fresh_client_for_domain_with_locale(
&self,
domain: &str,
locale: Option<&str>,
) -> Client
pub fn fresh_client_for_domain_with_locale( &self, domain: &str, locale: Option<&str>, ) -> Client
Get a fresh client that forces a new residential IP on the next request.
Use when a previous request was blocked (CAPTCHA, 403, empty SERP).
The fresh client has no connection pool, so DataImpulse assigns a new
residential IP. Falls back to client_for_domain_with_locale if not residential.
Sourcepub fn config(&self) -> &FetchiumConfig
pub fn config(&self) -> &FetchiumConfig
Get the config reference.
Sourcepub fn extract_domain(url: &str) -> String
pub fn extract_domain(url: &str) -> String
Extract domain from a URL for rate limiting.
Sourcepub async fn fetch(&self, url: &str) -> FetchiumResult<FetchResult>
pub async fn fetch(&self, url: &str) -> FetchiumResult<FetchResult>
Fetch a URL with retries, rate limiting, and size enforcement. Returns the response body as a string along with metadata.
Sourcepub async fn fetch_text(&self, url: &str) -> FetchiumResult<String>
pub async fn fetch_text(&self, url: &str) -> FetchiumResult<String>
Convenience: fetch a URL and return just the body text.
Sourcepub fn proxy_pool(&self) -> Option<&ProxyPool>
pub fn proxy_pool(&self) -> Option<&ProxyPool>
Get the proxy pool (if configured).
Sourcepub async fn fetch_via_proxy(&self, url: &str) -> FetchiumResult<FetchResult>
pub async fn fetch_via_proxy(&self, url: &str) -> FetchiumResult<FetchResult>
Fetch a URL through a rotating proxy. Falls back to direct if no proxy available.
Sourcepub async fn fetch_text_via_proxy(&self, url: &str) -> FetchiumResult<String>
pub async fn fetch_text_via_proxy(&self, url: &str) -> FetchiumResult<String>
Convenience: fetch text through a proxy.
Sourcepub async fn fetch_text_once(&self, url: &str) -> FetchiumResult<String>
pub async fn fetch_text_once(&self, url: &str) -> FetchiumResult<String>
Single-shot fetch — no retries, no rate limiting.
Use this for sources where connection errors are expected (e.g. third-party Piped/Invidious instances that may be down). A single fast attempt is better than burning 3.5s on retry sleeps when we’re racing many sources in parallel.
Sourcepub async fn post_json_once(
&self,
url: &str,
body: String,
) -> FetchiumResult<String>
pub async fn post_json_once( &self, url: &str, body: String, ) -> FetchiumResult<String>
Single-shot POST — no retries, no rate limiting.
Like fetch_text_once but sends a JSON POST body. Used for YouTube Innertube
API calls where each call is time-sensitive and retries are not desired.
Sourcepub async fn post_json(&self, url: &str, body: &str) -> FetchiumResult<String>
pub async fn post_json(&self, url: &str, body: &str) -> FetchiumResult<String>
POST JSON with retry support (uses the standard retry/rate-limit pipeline).
Sourcepub async fn post_json_with_header(
&self,
url: &str,
body: &str,
header_name: &str,
header_value: &str,
) -> FetchiumResult<String>
pub async fn post_json_with_header( &self, url: &str, body: &str, header_name: &str, header_value: &str, ) -> FetchiumResult<String>
POST JSON with a custom header (e.g., API key in non-standard header).
Trait Implementations§
Auto Trait Implementations§
impl !RefUnwindSafe for HttpClient
impl !UnwindSafe for HttpClient
impl Freeze for HttpClient
impl Send for HttpClient
impl Sync for HttpClient
impl Unpin for HttpClient
impl UnsafeUnpin for HttpClient
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> Downcast for Twhere
T: Any,
impl<T> Downcast for Twhere
T: Any,
Source§fn into_any(self: Box<T>) -> Box<dyn Any>
fn into_any(self: Box<T>) -> Box<dyn Any>
Box<dyn Trait> (where Trait: Downcast) to Box<dyn Any>. Box<dyn Any> can
then be further downcast into Box<ConcreteType> where ConcreteType implements Trait.Source§fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>
fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>
Rc<Trait> (where Trait: Downcast) to Rc<Any>. Rc<Any> can then be
further downcast into Rc<ConcreteType> where ConcreteType implements Trait.Source§fn as_any(&self) -> &(dyn Any + 'static)
fn as_any(&self) -> &(dyn Any + 'static)
&Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot
generate &Any’s vtable from &Trait’s.Source§fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
&mut Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot
generate &mut Any’s vtable from &mut Trait’s.Source§impl<T> DowncastSync for T
impl<T> DowncastSync for T
impl<T> Fruit for T
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more