Struct Spider 

pub struct Spider {
    pub api_key: String,
    pub client: Client,
}

Represents a Spider with API key and HTTP client.

Fields

api_key: String

The Spider API key.

client: Client

The Spider Client to re-use.

Implementations

impl Spider

pub fn new(api_key: Option<String>) -> Result<Self, &'static str>

Creates a new instance of Spider.

Arguments

• api_key - An optional API key. Defaults to using the ‘SPIDER_API_KEY’ env variable.

Returns

A new instance of Spider or an error string if no API key is provided.

pub fn new_with_client( api_key: Option<String>, client: Client, ) -> Result<Self, &'static str>

Creates a new instance of Spider.

Arguments

• api_key - An optional API key. Defaults to using the ‘SPIDER_API_KEY’ env variable.
• client - A custom client to pass in.

Returns

A new instance of Spider or an error string if no API key is provided.

pub async fn api_post( &self, endpoint: &str, data: impl Serialize + Debug + Clone + Send + Sync, content_type: &str, ) -> Result<Response, Error>

Sends a POST request to the API.

Arguments

• endpoint - The API endpoint.
• data - The request data as a HashMap.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API.

pub async fn api_get<T: Serialize>( &self, endpoint: &str, query_params: Option<&T>, ) -> Result<Value, Error>

Sends a GET request to the API.

Arguments

• endpoint - The API endpoint.

Returns

The response from the API as a JSON value.

pub async fn api_delete( &self, endpoint: &str, params: Option<HashMap<String, Value>>, ) -> Result<Response, Error>

Sends a DELETE request to the API.

Arguments

• endpoint - The API endpoint.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API.

pub async fn scrape_url( &self, url: &str, params: Option<RequestParams>, content_type: &str, ) -> Result<Value, Error>

Scrapes a URL.

Arguments

• url - The URL to scrape.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API as a JSON value.

pub async fn multi_scrape_url( &self, params: Option<Vec<RequestParams>>, content_type: &str, ) -> Result<Value, Error>

Scrapes multi URLs.

Arguments

• url - The URL to scrape.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API as a JSON value.

pub async fn crawl_url( &self, url: &str, params: Option<RequestParams>, stream: bool, content_type: &str, callback: Option<impl Fn(Value) + Send>, ) -> Result<Value, Error>

Crawls a URL.

Arguments

• url - The URL to crawl.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.
• callback - Optional callback function to handle each streamed chunk.

Returns

The response from the API as a JSON value.

pub async fn multi_crawl_url( &self, params: Option<Vec<RequestParams>>, stream: bool, content_type: &str, callback: Option<impl Fn(Value) + Send>, ) -> Result<Value, Error>

Crawls multiple URLs.

Arguments

• url - The URL to crawl.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.
• callback - Optional callback function to handle each streamed chunk.

Returns

The response from the API as a JSON value.

pub async fn links( &self, url: &str, params: Option<RequestParams>, _stream: bool, content_type: &str, ) -> Result<Value, Error>

Fetches links from a URL.

Arguments

• url - The URL to fetch links from.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API as a JSON value.

pub async fn multi_links( &self, params: Option<Vec<RequestParams>>, _stream: bool, content_type: &str, ) -> Result<Value, Error>

Fetches links from a URLs.

Arguments

• url - The URL to fetch links from.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API as a JSON value.

pub async fn screenshot( &self, url: &str, params: Option<RequestParams>, _stream: bool, content_type: &str, ) -> Result<Value, Error>

Takes a screenshot of a URL.

Arguments

• url - The URL to take a screenshot of.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API as a JSON value.

pub async fn multi_screenshot( &self, params: Option<Vec<RequestParams>>, _stream: bool, content_type: &str, ) -> Result<Value, Error>

Takes a screenshot of multiple URLs.

Arguments

• url - The URL to take a screenshot of.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API as a JSON value.

pub async fn search( &self, q: &str, params: Option<SearchRequestParams>, _stream: bool, content_type: &str, ) -> Result<Value, Error>

Searches for a query.

Arguments

• q - The query to search for.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API as a JSON value.

pub async fn multi_search( &self, params: Option<Vec<SearchRequestParams>>, content_type: &str, ) -> Result<Value, Error>

Searches for multiple querys.

Arguments

• q - The query to search for.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API as a JSON value.

pub async fn transform( &self, data: Vec<HashMap<&str, &str>>, params: Option<TransformParams>, _stream: bool, content_type: &str, ) -> Result<Value, Error>

Transforms data.

Arguments

• data - The data to transform.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

The response from the API as a JSON value.

pub async fn download( &self, url: Option<&str>, options: Option<HashMap<&str, i32>>, ) -> Result<Response, Error>

Download a record from storage.

Arguments

• url - Optional exact url of the file in storage.
• options - Optional options.
• stream - Whether streaming is enabled.

Returns

The response from the API.

pub async fn create_signed_url( &self, url: Option<&str>, options: Option<HashMap<&str, i32>>, ) -> Result<Value, Error>

Creates a signed URL of a file from storage.

Arguments

• url - Optional exact url of the file in storage.
• options - Optional options.
• stream - Whether streaming is enabled.

Returns

The response from the API.

pub async fn get_crawl_state( &self, url: &str, params: Option<RequestParams>, content_type: &str, ) -> Result<Value, Error>

Gets the crawl state of a URL.

Arguments

• url - The URL to get the crawl state of.
• params - Optional request parameters.
• stream - Whether streaming is enabled.
• content_type - The content type of the request.

Returns

pub async fn get_credits(&self) -> Result<Value, Error>

Get the account credits left.

pub async fn data_post( &self, table: &str, data: Option<RequestParams>, ) -> Result<Value, Error>

Send a request for a data record.

pub async fn query(&self, params: &QueryRequest) -> Result<Value, Error>

Query a record from the global DB.

pub async fn data_get( &self, table: &str, params: Option<RequestParams>, ) -> Result<Value, Error>

Get a table record.

pub async fn data_delete( &self, table: &str, params: Option<RequestParams>, ) -> Result<Value, Error>

Delete a record.

Trait Implementations

impl Debug for Spider

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Spider

fn default() -> Spider

Returns the “default value” for a type.