Crate spider_client 

The spider-client module provides the primary interface and functionalities for the Spider web crawler library, which is designed for rapid and efficient crawling of web pages to gather links using isolated contexts.

Features

• Multi-threaded Crawling: Spider can utilize multiple threads to parallelize the crawling process, drastically improving performance and allowing the ability to gather millions of pages in a short time.

• Configurable: The library provides various options to configure the crawling behavior, such as setting the depth of crawling, user-agent strings, delays between requests, and more.

• Link Gathering: One of the primary objectives of Spider is to gather and manage links from the web pages it crawls, compiling them into a structured format for further use.

Examples

Basic usage of the Spider client might look like this:

use spider_client::{Spider, RequestType, RequestParams};
use tokio;

#[tokio::main]
async fn main() {
    let spider = Spider::new(Some("myspiderapikey".into())).expect("API key must be provided");

    let url = "https://spider.cloud";

    // Scrape a single URL
    let scraped_data = spider.scrape_url(url, None, "application/json").await.expect("Failed to scrape the URL");

    println!("Scraped Data: {:?}", scraped_data);

    // Crawl a website
    let crawler_params = RequestParams {
        limit: Some(1),
        proxy_enabled: Some(true),
        metadata: Some(false),
        request: Some(RequestType::Http),
        ..Default::default()
    };

    let crawl_result = spider.crawl_url(url, Some(crawler_params), false, "application/json", None::<fn(serde_json::Value)>).await.expect("Failed to crawl the URL");

    println!("Crawl Result: {:?}", crawl_result);
}

Modules

• config: Contains the configuration options for the Spider client.
• utils: Utility functions used by the Spider client.

Re-exports

pub use shapes::request::*;

pub use shapes::response::*;

Modules

shapes

Structs

Spider

Represents a Spider with API key and HTTP client.

Functions

get_api_url

The API endpoint.

handle_csv

handle_json

Handle the json response.

handle_jsonl

Handle the jsonl response.

handle_text

Basic handle response to text

handle_xml

Handle the XML response.

parse_response