spider-middleware 0.3.4

Middleware implementations for the spider-lib web scraping framework.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

# spider-middleware

Built-in middleware implementations for `spider-lib` crawlers.

Most users enable middleware features from `spider-lib`. Use this crate directly when composing middleware in a custom runtime setup.

## Installation

```toml
[dependencies]
spider-middleware = "0.3.4"
```

## Built-in Middleware

Core (always available):

- `RateLimitMiddleware`
- `RetryMiddleware`
- `RefererMiddleware`

Optional (feature-gated):

- `middleware-cache` -> `HttpCacheMiddleware`
- `middleware-autothrottle` -> `AutoThrottleMiddleware`
- `middleware-proxy` -> `ProxyMiddleware`
- `middleware-user-agent` -> `UserAgentMiddleware`
- `middleware-robots` -> `RobotsTxtMiddleware`
- `middleware-cookies` -> `CookieMiddleware`

## Usage

```rust,ignore
use spider_middleware::{rate_limit::RateLimitMiddleware, retry::RetryMiddleware};

let crawler = spider_core::CrawlerBuilder::new(MySpider)
    .add_middleware(RateLimitMiddleware::default())
    .add_middleware(RetryMiddleware::new())
    .build()
    .await?;
```

## Feature Flags

- `core` (default)
- `middleware-cache`
- `middleware-autothrottle`
- `middleware-proxy`
- `middleware-user-agent`
- `middleware-robots`
- `middleware-cookies`

```toml
[dependencies]
spider-middleware = { version = "0.3.4", features = ["middleware-robots", "middleware-user-agent"] }
```

When using via `spider-lib`, enable root features with the same names.

## Related Crates

- [`spider-lib`](../README.md)
- [`spider-core`](../spider-core/README.md)
- [`spider-util`](../spider-util/README.md)

## License

MIT. See [LICENSE](../LICENSE).