spider 2.45.20

A web crawler and scraper, building blocks for data curation workloads.
Documentation

spider

There is very little structured metadata to build this page from currently. You should check the main library docs, readme, or Cargo.toml in case the author documented the features in them.

This version has 105 feature flags, 7 of them enabled by default.

default

basic (default)

io_uring (default)

basic_tls (default)

disk_native_tls (default)

reqwest_native_tls_native_roots (default)

disk (default)

adblock

agent

agent_chrome

agent_fs

agent_full

agent_openai

agent_search_bing

agent_search_brave

agent_search_serper

agent_search_tavily

agent_skills

agent_skills_s3

agent_webdriver

balance

cache

cache_chrome_hybrid

cache_chrome_hybrid_mem

cache_gemini

cache_mem

cache_openai

cache_request

chrome

chrome_cpu

chrome_headed

chrome_headless_new

chrome_intercept

chrome_remote_cache

chrome_remote_cache_disk

chrome_remote_cache_mem

chrome_screenshot

chrome_serde_stacker

chrome_simd

chrome_stealth

chrome_store_page

chrome_tls_connection

cmd

control

This feature flag does not enable additional features.

cookies

cowboy

This feature flag does not enable additional features.

cron

decentralized

decentralized_headers

disk_aws

encoding

This feature flag does not enable additional features.

extra_information

This feature flag does not enable additional features.

firewall

flexbuffers

fs

full_resources

This feature flag does not enable additional features.

gemini

glob

headers

hedge

This feature flag does not enable additional features.

inline-more

This feature flag does not enable additional features.

llm_json

openai

openai_slim_fit

This feature flag does not enable additional features.

page_error_status_details

This feature flag does not enable additional features.

real_browser

regex

This feature flag does not enable additional features.

remote_addr

This feature flag does not enable additional features.

reqwest_hickory_dns

reqwest_json

reqwest_multipart

reqwest_native_tls

reqwest_native_tls_alpn

reqwest_native_tls_manual_roots

reqwest_native_tls_vendored

reqwest_native_tls_webpki_roots

reqwest_rustls_tls

rquest_hickory_dns

search_bing

search_brave

search_serper

search_tavily

serde

simd

sitemap

smart

socks

spider_cloud

spoof

string_interner_bucket_backend

string_interner_buffer_backend

string_interner_string_backend

sync

time

This feature flag does not enable additional features.

tokio_io_std

tracing

ua_generator

webdriver

webdriver_chrome

webdriver_edge

webdriver_firefox

webdriver_headed

webdriver_screenshot

webdriver_stealth

wreq