Spider CLI

Is a command line tool to utilize the spider.

Dependencies

On Debian or other DEB based distributions:

$ sudo apt install openssl libssl-dev

On Fedora and other RPM based distributions:

$ sudo dnf install openssl-devel

Usage

The cli is a binary so do not add it to your cargo.toml file.

cargo install spider_cli

Cli

The following can also be ran via command line to run the crawler. All website options are available except website.on_link_find_callback.

spider --domain https://choosealicense.com --verbose true --delay 2000 --user_agent something@1.20