spider_cli 1.7.7

Multithreaded web crawler written in Rust.
spider_cli-1.7.7 is not a library.

Spider CLI

crate version

Is a command line tool to utilize the spider.

Dependencies

On Linux

  • OpenSSL 1.0.1, 1.0.2, 1.1.0, or 1.1.1

Usage

The CLI is a binary so do not add it to your Cargo.toml file.

cargo install spider_cli

Cli

The following can also be ran via command line to run the crawler. Website args are optional except domain.

spider --domain https://choosealicense.com --delay 2000 --blacklist-url license,books --user-agent something@1.20 crawl

All website options are available except website.on_link_find_callback.