Module proxy 

Proxy provider trait + baseline impl.

Substrate-level extension point for per-host proxy rotation. The engine calls ProxyProvider::next_proxy from reqwest::Proxy::custom per HTTP request, so implementations can rotate by host, by counter, or by external state. Returning None short-circuits to a direct connection.

Crawlberg ships StaticProxyProvider — a fixed-pool round-robin rotator. Cloud impls (e.g. BrightDataProxyProvider) plug in via crate::CrawlEngineBuilder::with_proxy_provider.

Browser-backend proxies (config.browser.proxy) are still configured at launch time via the static ProxyConfig value on CrawlConfig; the provider only routes the reqwest HTTP path. Mixing both is supported.

use std::sync::Arc;
use crawlberg::{ProxyConfig, ProxyProvider, StaticProxyProvider};

let pool = StaticProxyProvider::new(vec![
    ProxyConfig { url: "http://p1:8080".into(), username: None, password: None },
    ProxyConfig { url: "http://p2:8080".into(), username: None, password: None },
]);
let _arc: Arc<dyn ProxyProvider> = Arc::new(pool);

Structs

StaticProxyProvider

Round-robin pool of statically-configured proxies. Baseline impl shipped with crawlberg.

Traits

ProxyProvider

Resolves a ProxyConfig for an outbound HTTP request.