Crate spider_util 

spider-util

Utility types and traits for the spider-lib framework.

Example

use spider_util::{request::Request, item::ScrapedItem};
use url::Url;

let url = Url::parse("https://example.com").unwrap();
let request = Request::new(url);

#[spider_macro::scraped_item]
struct Article {
    title: String,
    content: String,
}

Modules

bloom_filter

Bloom Filter Module

error

Custom error types for the spider-core framework.

item

Data structures for scraped items in spider-lib.

request

Data structures for representing HTTP requests in spider-lib.

response

Data structures and utilities for handling HTTP responses in spider-lib.

response_type

Response types for the spider-lib framework.

selector_cache

Selector Cache Module

stream_response

Stream response implementation for memory-efficient web scraping.

utils

Utility functions for the spider-lib framework.