spider_cli 1.7.14

Multithreaded web crawler written in Rust.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# Spider CLI

![crate version](https://img.shields.io/crates/v/spider.svg)

Is a command line tool to utilize the spider.

## Dependencies

On Linux

- OpenSSL 1.0.1, 1.0.2, 1.1.0, or 1.1.1

## Usage

The CLI is a binary so do not add it to your `Cargo.toml` file.

```sh
cargo install spider_cli
```

## Cli

The following can also be ran via command line to run the crawler.
Website args are optional except `domain`. If you need verbose output pass in the `-v` flag.

```sh
spider -v --domain https://choosealicense.com crawl
```

Crawl and output all links visited on finished to a file.

```sh
spider  --domain https://choosealicense.com crawl -o > spider_choosealicense.json
```

All website options are available except `website.on_link_find_callback`.