spire-reqwest

Check out other spire projects here.

Reqwest-based HTTP client backend for Spire.

This crate provides [ReqwestBackend], an HTTP backend implementation that integrates with the Spire web scraping framework using the popular reqwest library.

Overview

spire-reqwest provides an HTTP client backend implementation for Spire on top of the popular reqwest library. This backend enables high-performance HTTP-based web scraping with support for cookies, headers, redirects, and other HTTP features.

ReqwestBackend implements Backend, and ReqwestSession implements Session, from spire-core: requests share the backend's connection pool.

Features

Built on Reqwest: Leverages the popular reqwest HTTP client library
Spire Integration: Seamlessly integrates with the Spire web scraping framework
Async/Await: Fully async implementation for better performance
TLS Support: Choose between rustls (default) and native TLS implementations

Usage

This crate is typically not used directly. Instead, enable the reqwest feature in the main spire crate and refer to the spire documentation for usage examples and guides.

[dependencies]
spire = { version = "0.9.0", features = ["reqwest"] }

See the book’s Feature flags page for configuration.

For advanced usage and custom configurations, see the API documentation.

Advanced Usage Examples

Configuring the client

ReqwestBackend::from_builder takes a function that makes the reqwest configuration, which a crawl with identities builds one client per identity from:

use spire_reqwest::ReqwestBackend;
use std::time::Duration;

let backend = ReqwestBackend::from_builder(|| {
    reqwest::Client::builder()
        .timeout(Duration::from_secs(30))
        .user_agent("MyBot/1.0")
})?;
# Ok::<_, spire_core::Error>(())

Identities

With Crawler::with_identities, each identity's requests go through a reqwest client of its own, with the identity's proxy and, with the cookies feature, its own cookie store. SOCKS proxies need the socks feature. See the identities chapter of the book.

Error Handling

Failed transfers are classified so the crawler retries only what may succeed next time: timeouts and connection errors are retried, invalid requests and redirect loops are not.

Error statuses follow the client's StatusPolicy. By default, 408, 429 and 5xx become retryable errors, with 429 and 503 pausing the request's host for as long as Retry-After asks, and other 4xx become errors that are not retried. StatusPolicy::Accept returns every response to the handler instead:

use spire_reqwest::{ReqwestBackend, StatusPolicy};

let backend = ReqwestBackend::new().with_status_policy(StatusPolicy::Accept);

Responses record the URL they were served from after redirects, available through spire_core::http::ResponseExt::final_uri.

Performance Considerations

The reqwest client maintains connection pools automatically.
Cookies and session state are maintained per reqwest client instance.
Consider reusing ReqwestBackend instances when possible to benefit from connection pooling.

Contributing

This crate follows the same contribution guidelines as the main spire project. Please see CONTRIBUTING.md for details.

License

This project is licensed under the MIT License; see the LICENSE file for details.

spire-reqwest 0.9.0