spider_worker 1.22.8

The fastest web crawler CLI written in Rust.
spider_worker-1.22.8 is not a library.

Spider Worker

crate version

A spider worker to decentralize the crawl lifting.

Dependencies

This project depends on the spider crate.

Usage

cargo run

WIP

  1. request handling io load
  2. parse links per domain