Struct HttpClientConfig 

pub struct HttpClientConfig {
    pub timeout: Duration,
    pub max_redirects: usize,
    pub retry_policy: RetryPolicy,
    pub user_agent: Arc<UserAgentRotator>,
    pub max_body_size: usize,
    pub pool_max_idle_per_host: usize,
    pub pool_max_idle: usize,
    pub tcp_keepalive: Option<Duration>,
}

Configuration for the HTTP client.

Controls timeout, redirect policy, retry behavior, connection pooling, and HTTP/2 settings. Can be constructed from a CrawlConfig.

Examples

use crawlkit_engine::{CrawlConfig, http::HttpClientConfig};

let config = HttpClientConfig::from(&CrawlConfig::default());
assert_eq!(config.max_body_size, 10 * 1024 * 1024);

Fields

timeout: Duration

Request timeout.

max_redirects: usize

Maximum number of redirects to follow.

retry_policy: RetryPolicy

Retry policy.

user_agent: Arc<UserAgentRotator>

User-agent rotator.

max_body_size: usize

Maximum response body size in bytes (0 = unlimited).

pool_max_idle_per_host: usize

Maximum number of idle connections per host.

pool_max_idle: usize

Maximum number of idle connections across all hosts.

tcp_keepalive: Option<Duration>

Whether to enable TCP keepalive.

Trait Implementations

impl Clone for HttpClientConfig

fn clone(&self) -> HttpClientConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HttpClientConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&CrawlConfig> for HttpClientConfig

fn from(config: &CrawlConfig) -> Self

Converts to this type from the input type.