spider 1.94.1

The fastest web crawler written in Rust.
Documentation

spider

There is very little structured metadata to build this page from currently. You should check the main library docs, readme, or Cargo.toml in case the author documented the features in them.

This version has 43 feature flags, 4 of them enabled by default.

default

  • sync
  • reqwest/native-tls-alpn
  • cookies
  • ua_generator

sync

  • tokio/sync
  • tokio/io-std

cookies

  • reqwest/cookies

ua_generator

  • dep:ua_generator

openai

  • chrome
  • serde
  • chrome_intercept
  • dep:async-openai
  • dep:tiktoken-rs
  • dep:lol_html
  • dep:serde_json

cache_chrome_hybrid

  • cache
  • chrome
  • dep:http-cache-semantics
  • dep:http-cache
  • dep:http

cron

  • dep:async_job
  • dep:chrono
  • dep:cron
  • dep:async-trait

serde

  • dep:serde
  • hashbrown/serde
  • compact_str/serde

cache

  • dep:reqwest-middleware
  • dep:http-cache-reqwest

smart

  • chrome
  • dep:regex

decentralized_headers

  • dep:const_format
  • dep:itertools

glob

  • dep:regex
  • dep:itertools

napi_rustls_tls

  • napi
  • reqwest/rustls-tls

decentralized

  • serde
  • flexbuffers

chrome_store_page

  • chrome
  • serde

cache_mem

  • cache
  • http-cache-reqwest/manager-moka

flexbuffers

  • dep:flexbuffers

regex

  • dep:regex

cache_openai

  • dep:moka

encoding

  • dep:encoding_rs

jemalloc

  • tikv-jemallocator

chrome_intercept

  • chrome

chrome_stealth

  • chrome

tikv-jemallocator

  • dep:tikv-jemallocator

chrome_screenshot

  • chrome

chrome_headless_new

  • chrome

fs

  • tokio/fs

chrome_cpu

  • chrome

sitemap

  • dep:sitemap

socks

  • reqwest/socks

napi

  • dep:napi

js

  • dep:jsdom

reqwest_json

  • reqwest/json

chrome_headed

  • chrome

spoof

  • dep:fastrand

chrome

  • dep:chromiumoxide

control

    This feature flag does not enable additional features.

time

    This feature flag does not enable additional features.

openai_slim_fit

    This feature flag does not enable additional features.

real_browser

    This feature flag does not enable additional features.

headers

    This feature flag does not enable additional features.

full_resources

    This feature flag does not enable additional features.

budget

    This feature flag does not enable additional features.