spider-1.94.2
- spider 1.94.2
- Docs.rs crate page
- MIT
- Links
- Documentation
- Repository
- crates.io
- Source
- Owners
- madeindjs
- j-mendez
- Dependencies
- ahash ^0.8.11 normal
  async-openai ^0.20.0 normal optional
  async-trait ^0.1.75 normal optional
  async_job ^0.1.4 normal optional
  bytes ^1.5.0 normal
  case_insensitive_string ^0.2.2 normal
  chromiumoxide ^0.5.7 normal optional
  chrono ^0.4.31 normal optional
  compact_str ^0.7.1 normal
  const_format ^0.2.32 normal optional
  cron ^0.12.0 normal optional
  cssparser ^0.31.2 normal
  ego-tree ^0.6.2 normal
  encoding_rs ^0.8.33 normal optional
  fast_html5ever ^0.26.4 normal
  fastrand ^2 normal optional
  flexbuffers ^2.0.0 normal optional
  hashbrown ^0.14.3 normal
  http ^1 normal optional
  http-cache ^0.19.0 normal optional
  http-cache-reqwest ^0.14.0 normal optional
  http-cache-semantics ^2 normal optional
  itertools ^0.12.0 normal optional
  jsdom ^0.0.11-alpha.1 normal optional
  lazy_static ^1.4.0 normal
  log ^0.4 normal
  lol_html ^1.2.1 normal optional
  moka ^0.12.1 normal optional
  napi ^2 normal optional
  num_cpus ^1.16.0 normal
  percent-encoding ^2.3.1 normal
  quick-xml ^0.31.0 normal
  regex ^1.10.4 normal optional
  reqwest ^0.12 normal
  reqwest-middleware ^0.3.0 normal optional
  selectors ^0.25.0 normal
  serde ^1 normal optional
  serde_json ^1 normal optional
  sitemap ^0.4.1 normal optional
  smallvec ^1.11.2 normal
  string_concat ^0.0.1 normal
  strum ^0.26.2 normal
  tendril ^0.4.3 normal
  tiktoken-rs ^0.5.8 normal optional
  tokio ^1 normal
  tokio-stream ^0.1.14 normal
  ua_generator ^0.4.2 normal optional
  url ^2 normal
  tikv-jemallocator ^0.5.0 normal optional
- Versions
- 100% of the crate is documented
Go to latest version
Platform
Feature flags

Rust

spider1.94.2

Module robotparser

Modules

In spider::packages

Module spider::packages::robotparser

Expand description

robot parser

Modules§

parser
robots.txt parser for Rust.