Module proxy 

Proxy configuration and rotation for HTTP requests.

When scraping at scale, routing requests through proxy servers helps avoid IP-based rate limiting and bans. This module provides two mechanisms:

• Static proxy – A single Proxy set on FetcherConfig that is used for every request.
• Proxy rotation – A ProxyRotator that cycles through a pool of proxies, picking the next one for each request according to a RotationStrategy.

The is_proxy_error helper function inspects error messages to determine whether a failure was proxy-related, which is useful for deciding whether to retry with a different proxy.

Structs

ProxyRotator

Thread-safe proxy rotator that cycles through a list of proxies using a configurable strategy.

Enums

Proxy

A proxy server specification, either as a URL string or a structured configuration.

Functions

cyclic_rotation

The default rotation strategy that cycles through proxies sequentially (0, 1, 2, …, then back to 0). The returned index is wrapped with modulo in ProxyRotator::get_proxy, so this simply returns current + 1.

is_proxy_error

Returns true if the error message indicates a proxy-related failure. This checks for common proxy error patterns like “connection refused”, “tunnel failed”, etc. Useful in retry logic to decide whether to switch to a different proxy.

Type Aliases

RotationStrategy

A function that determines the next proxy index given the proxy list and the current index. Implement your own to create custom rotation strategies (e.g., random, weighted, or geo-aware selection).