spider 2.38.120

A web crawler and scraper, building blocks for data curation workloads.
Documentation

spider

There is very little structured metadata to build this page from currently. You should check the main library docs, readme, or Cargo.toml in case the author documented the features in them.

This version has 78 feature flags, 7 of them enabled by default.

default

basic (default)

io_uring (default)

basic_tls (default)

disk_native_tls (default)

reqwest_native_tls_native_roots (default)

disk (default)

adblock

balance

cache

cache_chrome_hybrid

cache_chrome_hybrid_mem

cache_gemini

cache_mem

cache_openai

cache_request

chrome

chrome_cpu

chrome_headed

chrome_headless_new

chrome_intercept

chrome_remote_cache

chrome_remote_cache_disk

chrome_remote_cache_mem

chrome_screenshot

chrome_serde_stacker

chrome_simd

chrome_stealth

chrome_store_page

chrome_tls_connection

cmd

control

This feature flag does not enable additional features.

cookies

cowboy

This feature flag does not enable additional features.

cron

decentralized

decentralized_headers

disk_aws

encoding

This feature flag does not enable additional features.

extra_information

This feature flag does not enable additional features.

firewall

flexbuffers

fs

full_resources

This feature flag does not enable additional features.

gemini

glob

headers

inline-more

This feature flag does not enable additional features.

openai

openai_slim_fit

This feature flag does not enable additional features.

page_error_status_details

This feature flag does not enable additional features.

real_browser

regex

This feature flag does not enable additional features.

remote_addr

This feature flag does not enable additional features.

reqwest_hickory_dns

reqwest_json

reqwest_multipart

reqwest_native_tls

reqwest_native_tls_alpn

reqwest_native_tls_manual_roots

reqwest_native_tls_vendored

reqwest_native_tls_webpki_roots

reqwest_rustls_tls

rquest_hickory_dns

serde

simd

sitemap

smart

socks

spoof

string_interner_bucket_backend

string_interner_buffer_backend

string_interner_string_backend

sync

time

This feature flag does not enable additional features.

tokio_io_std

tracing

ua_generator

wreq