Crate spider_middleware 

spider-middleware

Built-in middleware implementations for the spider-lib framework.

Provides middlewares for rate limiting, retries, user-agent rotation, cookies, and more.

Example

use spider_middleware::rate_limit::RateLimitMiddleware;
use spider_middleware::retry::RetryMiddleware;

let crawler = CrawlerBuilder::new(MySpider)
    .add_middleware(RateLimitMiddleware::default())
    .add_middleware(RetryMiddleware::new())
    .build()
    .await?;

Modules

middleware

Core Middleware trait and related types for the spider-core framework.

prelude

Commonly used items from the spider-middleware crate.

rate_limit

Rate Limit Middleware for controlling request frequency.

referer

Referer Middleware for managing HTTP Referer headers.

retry

Retry Middleware for handling failed requests.

Structs

Request

An HTTP request to be processed by the crawler.

Response

Represents an HTTP response received from a server.

Enums

Body

The body of an HTTP request.