Module request 

Data structures for representing HTTP requests in spider-lib.

This module defines the Request struct, which is a central component for constructing and managing outgoing HTTP requests within the spider-lib framework. It encapsulates all necessary details of an HTTP request, including:

• The target URL and HTTP method
• Request headers and an optional request body (supporting JSON, form data, or raw bytes)
• Metadata for tracking retry attempts or other custom information

Additionally, the module provides methods for building requests, incrementing retry counters, and generating unique fingerprints for request deduplication and caching.

Example

use spider_util::request::{Request, Body};
use url::Url;
use serde_json::json;

// Create a simple GET request
let url = Url::parse("https://example.com").unwrap();
let request = Request::new(url);

// Create a POST request with JSON body
let post_request = Request::new(Url::parse("https://api.example.com/data").unwrap())
    .with_method(reqwest::Method::POST)
    .with_json(json!({"key": "value"}));

Structs

Request

An HTTP request to be processed by the crawler.

Enums

Body

The body of an HTTP request.