reqwest-proxy-pool 0.2.1

proxy pool middleware for reqwest
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123

# reqwest-proxy-pool

Proxy pool middleware implementation for [`reqwest-middleware`](https://crates.io/crates/reqwest-middleware).

[![Crates.io](https://img.shields.io/crates/v/reqwest-proxy-pool.svg)](https://crates.io/crates/reqwest-proxy-pool)
[![Docs.rs](https://docs.rs/reqwest-proxy-pool/badge.svg)](https://docs.rs/reqwest-proxy-pool)
[![CI](https://github.com/suiwenfeng/reqwest-proxy-pool/actions/workflows/ci.yml/badge.svg)](https://github.com/suiwenfeng/reqwest-proxy-pool/actions/workflows/ci.yml)
[![Rust 1.85+](https://img.shields.io/badge/rust-1.85%2B-orange.svg)](https://www.rust-lang.org)

## Features

### ✨ Comprehensive Proxy Support

- Automatic parsing of free SOCKS5/SOCKS5H proxies from multiple sources
- Built-in health checking with customizable timeout and test URL

### ⚡ Intelligent Proxy Management

- Multiple proxy selection strategies (FastestResponse, RoundRobin, Random)
- Per-proxy rate limiting to avoid bans
- Automatic retry mechanism for failed requests
- Custom response classifier for business-level proxy health (anti-bot/captcha detection)

### 🔧 Easy Configuration

- Simple builder pattern for configuration
- Seamless integration with reqwest middleware stack

## Quickstart

### Installation

Add to your `Cargo.toml`:

```toml
[dependencies]
reqwest = "0.13"
reqwest-proxy-pool = "0.2"
reqwest-middleware = "0.5"
tokio = { version = "1", features = ["full"] }
```

### Usage

```rust
use reqwest_middleware::ClientBuilder;
use reqwest_proxy_pool::{
    ProxyPoolConfig, ProxyPoolMiddleware, ProxyResponseVerdict, ProxySelectionStrategy,
    ResponseClassifier,
};
use std::time::Duration;

struct CaptchaDetector;

impl ResponseClassifier for CaptchaDetector {
    fn classify(&self, response: &reqwest::Response) -> ProxyResponseVerdict {
        match response.status().as_u16() {
            403 | 429 => ProxyResponseVerdict::ProxyBlocked,
            500..=599 => ProxyResponseVerdict::Passthrough,
            _ => ProxyResponseVerdict::Success,
        }
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ProxyPoolConfig::builder()
        .sources(vec![
            "https://cdn.jsdelivr.net/gh/dpangestuw/Free-Proxy@main/socks5_proxies.txt",
            "https://cdn.jsdelivr.net/gh/proxifly/free-proxy-list@main/proxies/protocols/socks5/data.txt",
        ])
        .health_check_timeout(Duration::from_secs(5))
        .health_check_url("https://httpbin.org/ip")
        .retry_count(2)
        .selection_strategy(ProxySelectionStrategy::FastestResponse)
        .max_requests_per_second(3.0)
        .response_classifier(CaptchaDetector)
        .danger_accept_invalid_certs(true)
        .build();

    let proxy_pool = ProxyPoolMiddleware::new(config).await?;

    let client = ClientBuilder::new(reqwest::Client::new())
        .with(proxy_pool)
        .build();

    let response = client.get("https://httpbin.org/ip").send().await?;
    println!("Status: {}", response.status());
    println!("Response: {}", response.text().await?);

    Ok(())
}
```

### Configuration Options

| Option                    | Description                           | Default                    |
|---------------------------|---------------------------------------|----------------------------|
| `sources`                 | List of URLs providing proxy lists    | Required                   |
| `health_check_interval`   | Interval for background health checks | 300s                       |
| `health_check_timeout`    | Timeout for proxy health checks       | 10s                        |
| `min_available_proxies`   | Min available proxies                 | 3                          |
| `health_check_url`        | URL to test proxy health              | `"https://www.google.com"` |
| `retry_count`             | Number of retries for failed requests | 3                          |
| `selection_strategy`      | Proxy selection algorithm             | `FastestResponse`          |
| `max_requests_per_second` | Rate limit per proxy                  | 5.0                        |
| `response_classifier`     | Custom response classifier for proxy health | `DefaultResponseClassifier` |
| `danger_accept_invalid_certs` | Accept invalid TLS certs (needed for most free proxies) | `false` |

## License

<sup>
Licensed under either of <a href="LICENSE-APACHE">Apache License, Version 2.0</a>
or <a href="LICENSE-MIT">MIT license</a> at your option.
</sup>

<br>

<sub>
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall be
dual licensed as above, without any additional terms or conditions.
</sub>