spire-reqwest 0.3.0

Reqwest-based HTTP client backend for Spire web scraping framework
Documentation

spire-reqwest

Build Status Crate Docs Crate Version Crate Coverage

Check out other spire projects here.

Reqwest-based HTTP client backend for Spire.

This crate provides [HttpClient], an HTTP backend implementation that integrates with the Spire web scraping framework using the popular reqwest library.

Overview

spire-reqwest provides an HTTP client backend implementation for Spire on top of the popular reqwest library. This backend enables high-performance HTTP-based web scraping with support for cookies, headers, redirects, and other HTTP features.

HttpClient implements both the Backend and Client traits from spire-core: every request of a crawl shares one connection pool.

Features

  • Built on Reqwest: Leverages the popular reqwest HTTP client library
  • Spire Integration: Seamlessly integrates with the Spire web scraping framework
  • Async/Await: Fully async implementation for better performance
  • TLS Support: Choose between rustls (default) and native TLS implementations

Usage

This crate is typically not used directly. Instead, enable the reqwest feature in the main spire crate and refer to the spire documentation for usage examples and guides.

[dependencies]
spire = { version = "0.3.0", features = ["reqwest"] }

Features

TLS through rustls, HTTP/2 and system proxies are on by default. Everything else is opt-in:

[dependencies]
spire-reqwest = { version = "0.3.0", features = ["gzip", "brotli", "cookies"] }

# Native TLS instead of rustls
spire-reqwest = { version = "0.3.0", default-features = false, features = ["native-tls", "http2"] }
Feature Enables
rustls-tls (default), native-tls TLS implementation
http2 (default) HTTP/2 negotiation
system-proxy (default) Proxies from the environment and system settings
gzip, brotli, zstd, deflate Transparent response decompression
cookies A cookie store shared by the whole crawl
socks SOCKS5 proxies
hickory-dns DNS through hickory instead of the system resolver

For advanced usage and custom configurations, see the API documentation.

Advanced Usage Examples

Creating from reqwest Client

use spire_reqwest::HttpClient;
use std::time::Duration;

let reqwest_client = reqwest::Client::builder()
    .timeout(Duration::from_secs(30))
    .user_agent("MyBot/1.0")
    .build()
    .unwrap();

let backend = HttpClient::from(reqwest_client);

Error Handling

Failed transfers are classified so the crawler retries only what may succeed next time: timeouts and connection errors are retried, invalid requests and redirect loops are not.

Error statuses follow the client's StatusPolicy. By default, 408, 429 and 5xx become retryable errors, with 429 and 503 pausing the request's tag for as long as Retry-After asks, and other 4xx become errors that are not retried. StatusPolicy::Accept returns every response to the handler instead:

use spire_reqwest::{HttpClient, StatusPolicy};

let backend = HttpClient::new().with_status_policy(StatusPolicy::Accept);

Responses record the URL they were served from after redirects, available through spire_core::request::ResponseExt::final_uri.

Performance Considerations

  • The reqwest client maintains connection pools automatically.
  • Cookies and session state are maintained per reqwest client instance.
  • Consider reusing HttpClient instances when possible to benefit from connection pooling.

Contributing

This crate follows the same contribution guidelines as the main spire project. Please see CONTRIBUTING.md for details.

License

This project is licensed under the MIT License; see the LICENSE file for details.