Skip to main content

HttpClient

Struct HttpClient 

Source
pub struct HttpClient { /* private fields */ }
Expand description

Shared HTTP client with connection pooling, retries, and rate limiting.

Implementations§

Source§

impl HttpClient

Source

pub fn new(config: &FetchiumConfig) -> FetchiumResult<Self>

Create a new HTTP client from config.

Source

pub fn client(&self) -> &Client

Get the inner reqwest client for direct use.

Source

pub fn client_for_domain_with_locale( &self, domain: &str, locale: Option<&str>, ) -> Client

Get a reqwest client for a specific domain.

Priority order:

  1. DataImpulse residential proxy (country-targeted) — Google only
  2. Direct connection — default path for everything else

Cheap path by default: non-Google traffic stays direct until a backend explicitly asks for proxy escalation.

Source

pub fn client_for_domain(&self, domain: &str) -> Client

Backwards-compatible wrapper — no locale hint.

Source

pub fn client_direct(&self) -> Client

Direct client — bypasses ALL proxy routing (residential and datacenter).

Use for first-attempt requests where proxy is not yet needed. If the direct attempt fails, escalate to client_for_domain_with_locale. Saves DataImpulse GB when direct connections succeed (majority of requests).

Source

pub fn proxy_client_for_domain_with_locale( &self, domain: &str, locale: Option<&str>, ) -> Client

Get a forced proxy client for a domain.

Google uses residential. All other proxy-eligible domains use the fixed proxy pool. Falls back to direct when no proxy is available.

Source

pub fn fresh_client_for_domain_with_locale( &self, domain: &str, locale: Option<&str>, ) -> Client

Get a fresh client that forces a new residential IP on the next request.

Use when a previous request was blocked (CAPTCHA, 403, empty SERP). The fresh client has no connection pool, so DataImpulse assigns a new residential IP. Falls back to client_for_domain_with_locale if not residential.

Source

pub fn config(&self) -> &FetchiumConfig

Get the config reference.

Source

pub fn extract_domain(url: &str) -> String

Extract domain from a URL for rate limiting.

Source

pub async fn fetch(&self, url: &str) -> FetchiumResult<FetchResult>

Fetch a URL with retries, rate limiting, and size enforcement. Returns the response body as a string along with metadata.

Source

pub async fn fetch_text(&self, url: &str) -> FetchiumResult<String>

Convenience: fetch a URL and return just the body text.

Source

pub fn proxy_pool(&self) -> Option<&ProxyPool>

Get the proxy pool (if configured).

Source

pub async fn fetch_via_proxy(&self, url: &str) -> FetchiumResult<FetchResult>

Fetch a URL through a rotating proxy. Falls back to direct if no proxy available.

Source

pub async fn fetch_text_via_proxy(&self, url: &str) -> FetchiumResult<String>

Convenience: fetch text through a proxy.

Source

pub async fn fetch_text_once(&self, url: &str) -> FetchiumResult<String>

Single-shot fetch — no retries, no rate limiting.

Use this for sources where connection errors are expected (e.g. third-party Piped/Invidious instances that may be down). A single fast attempt is better than burning 3.5s on retry sleeps when we’re racing many sources in parallel.

Source

pub async fn post_json_once( &self, url: &str, body: String, ) -> FetchiumResult<String>

Single-shot POST — no retries, no rate limiting.

Like fetch_text_once but sends a JSON POST body. Used for YouTube Innertube API calls where each call is time-sensitive and retries are not desired.

Source

pub async fn post_json(&self, url: &str, body: &str) -> FetchiumResult<String>

POST JSON with retry support (uses the standard retry/rate-limit pipeline).

Source

pub async fn post_json_with_header( &self, url: &str, body: &str, header_name: &str, header_value: &str, ) -> FetchiumResult<String>

POST JSON with a custom header (e.g., API key in non-standard header).

Trait Implementations§

Source§

impl Clone for HttpClient

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> Downcast for T
where T: Any,

Source§

fn into_any(self: Box<T>) -> Box<dyn Any>

Convert Box<dyn Trait> (where Trait: Downcast) to Box<dyn Any>. Box<dyn Any> can then be further downcast into Box<ConcreteType> where ConcreteType implements Trait.
Source§

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

Convert Rc<Trait> (where Trait: Downcast) to Rc<Any>. Rc<Any> can then be further downcast into Rc<ConcreteType> where ConcreteType implements Trait.
Source§

fn as_any(&self) -> &(dyn Any + 'static)

Convert &Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot generate &Any’s vtable from &Trait’s.
Source§

fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)

Convert &mut Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot generate &mut Any’s vtable from &mut Trait’s.
Source§

impl<T> DowncastSync for T
where T: Any + Send + Sync,

Source§

fn into_any_arc(self: Arc<T>) -> Arc<dyn Any + Sync + Send> ⓘ

Convert Arc<Trait> (where Trait: Downcast) to Arc<Any>. Arc<Any> can then be further downcast into Arc<ConcreteType> where ConcreteType implements Trait.
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T> Fruit for T
where T: Send + Downcast,

Source§

impl<T> Instrument for T

Source§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Source§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> Pointable for T

Source§

const ALIGN: usize

The alignment of pointer.
Source§

type Init = T

The type for initializers.
Source§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
Source§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
Source§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
Source§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
Source§

impl<T> PolicyExt for T
where T: ?Sized,

Source§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow only if self and other return Action::Follow. Read more
Source§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V

Source§

impl<T> WithSubscriber for T

Source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more