spider 1.93.21

The fastest web crawler written in Rust.
Documentation

spider

There is very little structured metadata to build this page from currently. You should check the main library docs, readme, or Cargo.toml in case the author documented the features in them.

This version has 41 feature flags, 3 of them enabled by default.

default

  • sync
  • reqwest/native-tls-alpn
  • cookies

sync

  • tokio/sync
  • tokio/io-std

cookies

  • reqwest/cookies

openai

  • chrome
  • serde
  • chrome_intercept
  • dep:async-openai
  • dep:tiktoken-rs
  • dep:lol_html
  • dep:serde_json

cron

  • dep:async_job
  • dep:chrono
  • dep:cron
  • dep:async-trait

serde

  • dep:serde
  • hashbrown/serde
  • compact_str/serde

napi_rustls_tls

  • napi
  • reqwest/rustls-tls

smart

  • chrome
  • dep:regex

cache_mem

  • cache
  • http-cache-reqwest/manager-moka

decentralized_headers

  • dep:const_format
  • dep:itertools

decentralized

  • serde
  • flexbuffers

glob

  • dep:regex
  • dep:itertools

cache

  • dep:reqwest-middleware
  • dep:http-cache-reqwest

chrome_store_page

  • chrome
  • serde

chrome_intercept

  • chrome

napi

  • dep:napi

chrome_screenshot

  • chrome

js

  • dep:jsdom

reqwest_json

  • reqwest/json

ua_generator

  • dep:ua_generator

chrome_stealth

  • chrome

chrome_headed

  • chrome

socks

  • reqwest/socks

chrome_headless_new

  • chrome

fs

  • tokio/fs

flexbuffers

  • dep:flexbuffers

cache_openai

  • dep:moka

regex

  • dep:regex

jemalloc

  • tikv-jemallocator

chrome

  • dep:chromiumoxide

sitemap

  • dep:sitemap

tikv-jemallocator

  • dep:tikv-jemallocator

encoding

  • dep:encoding_rs

chrome_cpu

  • chrome

control

    This feature flag does not enable additional features.

real_browser

    This feature flag does not enable additional features.

full_resources

    This feature flag does not enable additional features.

budget

    This feature flag does not enable additional features.

headers

    This feature flag does not enable additional features.

time

    This feature flag does not enable additional features.

openai_slim_fit

    This feature flag does not enable additional features.