Crate spider_middleware 

spider-middleware

Built-in middleware implementations for the spider-lib framework.

Provides middlewares for rate limiting, retries, user-agent rotation, cookies, and more.

Example

use spider_middleware::rate_limit::RateLimitMiddleware;
use spider_middleware::retry::RetryMiddleware;

let crawler = CrawlerBuilder::new(MySpider)
    .add_middleware(RateLimitMiddleware::default())
    .add_middleware(RetryMiddleware::new())
    .build()
    .await?;

Modules

middleware

Core Middleware trait and related types for the spider-core framework.

prelude

Commonly used items from the spider-middleware crate.

rate_limit

Rate Limit Middleware for controlling request frequency.

referer

Referer Middleware for managing HTTP Referer headers.

request

Data structures for representing HTTP requests in spider-lib.

retry

Retry Middleware for handling failed requests.

Structs

Request

Response

Represents an HTTP response received from a server.