Struct SpiderCloudConfig 

pub struct SpiderCloudConfig {
    pub api_key: String,
    pub mode: SpiderCloudMode,
    pub api_url: String,
    pub proxy_url: String,
    pub return_format: SpiderCloudReturnFormat,
    pub return_formats: Option<Vec<SpiderCloudReturnFormat>>,
    pub extra_params: Option<HashMap<String, Value>>,
}

Available on crate feature spider_cloud only.

Configuration for spider.cloud integration.

Spider Cloud provides anti-bot bypass, proxy rotation, and high-throughput data collection. Sign up at https://spider.cloud to obtain an API key.

Fields

api_key: String

API key / secret. Sign up at https://spider.cloud to get one.

mode: SpiderCloudMode

Integration mode.

api_url: String

API base URL (default: https://api.spider.cloud).

proxy_url: String

Proxy URL (default: https://proxy.spider.cloud).

return_format: SpiderCloudReturnFormat

Return format for API responses (default: SpiderCloudReturnFormat::Raw).

return_formats: Option<Vec<SpiderCloudReturnFormat>>

Request multiple return formats in a single crawl.

When set, the API returns content as an object keyed by format (e.g. {"markdown": "...", "raw": "..."}). The primary return_format is stored in Page::get_content and the extras are accessible via Page::get_content_for.

extra_params: Option<HashMap<String, Value>>

Extra params forwarded in API mode (e.g. stealth, fingerprint, cache).

Implementations

impl SpiderCloudConfig

pub fn new(api_key: impl Into<String>) -> Self

Create a new config with defaults (Proxy mode).

pub fn with_mode(self, mode: SpiderCloudMode) -> Self

Set the integration mode.

pub fn with_api_url(self, url: impl Into<String>) -> Self

Set a custom API base URL.

pub fn with_proxy_url(self, url: impl Into<String>) -> Self

Set a custom proxy URL.

pub fn with_return_format(self, fmt: impl Into<SpiderCloudReturnFormat>) -> Self

Set the return format for API responses.

Accepts SpiderCloudReturnFormat directly or a string like "markdown":

config.with_return_format(SpiderCloudReturnFormat::Markdown)
config.with_return_format("markdown")

pub fn with_return_formats(self, formats: Vec<SpiderCloudReturnFormat>) -> Self

Request multiple return formats in a single crawl.

The first format becomes the primary content (accessible via Page::get_content), and all formats are accessible via Page::get_content_for.

config.with_return_formats(vec![
    SpiderCloudReturnFormat::Markdown,
    SpiderCloudReturnFormat::Raw,
])

pub fn has_multiple_formats(&self) -> bool

Check if multiple return formats are requested.

pub fn with_extra_params(self, params: HashMap<String, Value>) -> Self

Set extra params for API mode.

pub fn should_fallback(&self, status_code: u16, body: Option<&[u8]>) -> bool

Determine if a response should trigger a spider.cloud API fallback.

This encapsulates the intelligence about which status codes and content patterns indicate the page needs spider.cloud’s help.

Checks for:

• HTTP 403 (Forbidden) — typically bot protection
• HTTP 429 (Too Many Requests) — rate limiting
• HTTP 503 (Service Unavailable) — often Cloudflare/DDoS protection
• HTTP 520-530 (Cloudflare error range)
• HTTP 5xx (server errors)
• Empty body on what should be an HTML page
• Known CAPTCHA / challenge page markers in the response body

pub fn fallback_route(&self) -> &'static str

Get the fallback API route for this config.

• Smart mode → /unblocker (best for bot-protected pages)
• Fallback mode → /crawl (general purpose)
• Other modes → /crawl (default)

pub fn uses_proxy(&self) -> bool

Whether this mode uses the proxy transport layer.

Trait Implementations

impl Clone for SpiderCloudConfig

fn clone(&self) -> SpiderCloudConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SpiderCloudConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SpiderCloudConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for SpiderCloudConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for SpiderCloudConfig

fn eq(&self, other: &SpiderCloudConfig) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for SpiderCloudConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Eq for SpiderCloudConfig

impl StructuralPartialEq for SpiderCloudConfig