Struct HttpClient 

pub struct HttpClient { /* private fields */ }

An HTTP client with retry, redirect tracking, and user-agent rotation.

Built on top of reqwest::Client with additional features for web crawling:

• Manual redirect following with hop recording
• Exponential backoff retry for transient failures
• User-agent rotation across requests
• Response body size limiting
• Streaming responses

Examples

use crawlkit_engine::{CrawlConfig, HttpClient};
use url::Url;

let client = HttpClient::from_crawl_config(&CrawlConfig::default())?;
let url = Url::parse("https://example.com")?;
let result = client.fetch(&url).await?;
assert_eq!(result.status_code, 200);

Implementations

impl HttpClient

pub fn new(config: HttpClientConfig) -> Result<Self, CrawlError>

Creates a new HttpClient from the given configuration.

Builds a reqwest::Client with TLS, HTTP/2 multiplexing, connection pooling, and redirect policy.

Errors

Returns CrawlError::RequestFailed if the underlying reqwest client cannot be built (e.g., invalid TLS configuration).

pub fn from_crawl_config(config: &CrawlConfig) -> Result<Self, CrawlError>

Creates a new HttpClient from a CrawlConfig.

Convenience method that converts the crawl config into an HttpClientConfig and builds the client.

Errors

Returns CrawlError::RequestFailed if the client cannot be built.

pub fn high_throughput(config: HttpClientConfig) -> Result<Self, CrawlError>

Creates a new HttpClient optimized for high-throughput crawling.

Enables HTTP/2, larger connection pools, and TCP keepalive.

pub async fn fetch(&self, url: &Url) -> Result<FetchResult, CrawlError>

Fetches a URL with retry logic and redirect tracking.

Returns a FetchResult with the final URL, status, headers, and body. Follows redirects manually to record each hop in the chain.

Errors

Returns CrawlError::RequestFailed on network errors after retries are exhausted, or CrawlError::TooManyRedirects if the redirect limit is exceeded.

pub async fn fetch_with_redirects( &self, url: &Url, max_hops: usize, ) -> Result<FetchResult, CrawlError>

Fetches a URL, following up to max_hops redirects manually.

Each redirect hop is recorded. If the hop limit is exceeded, CrawlError::TooManyRedirects is returned.

Errors

Returns errors for network failures or exceeded redirect limits.

pub fn inner(&self) -> &Client

Returns a reference to the inner reqwest::Client.

pub fn config(&self) -> &HttpClientConfig

Returns a reference to the client configuration.

pub async fn fetch_stream<F>( &self, url: &Url, on_chunk: F, ) -> Result<FetchResult, CrawlError>

where F: FnMut(&str) + Send,

Fetches a URL and streams the response body, calling the callback with each chunk.

This is useful for large pages where you want to process HTML as it arrives rather than buffering the entire response in memory.

Errors

Returns errors for network failures or redirect limit exceeded.

pub async fn fetch_reader( &self, url: &Url, ) -> Result<FetchStreamReader, CrawlError>

Fetches a URL and returns the response as a streaming reader.

Returns the response metadata (status, headers) and a streaming body. The caller can read chunks from the stream via FetchStreamReader::next_chunk.

Errors

Returns errors for network failures.