spider 1.2.0

Multithreaded Web spider crawler written in Rust.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

# Spider

![crate version](https://img.shields.io/crates/v/spider.svg)

Multithreaded Web spider crawler written in Rust.

## Dependencies

~~~bash
$ apt install openssl libssl-dev
~~~

## Usage

Add this dependency to your _Cargo.toml_ file.

~~~toml
[dependencies]
spider = "1.0.2"
~~~

and then you'll be able to use library. Here a simple example

~~~rust
extern crate spider;

use spider::website::Website;

fn main() {
    let mut website: Website = Website::new("https://choosealicense.com");
    website.crawl();

    for page in website.get_pages() {
        println!("- {}", page.get_url());
    }
}
~~~

You can use `Configuration` object to configure your crawler:

~~~rust
// ..
let mut website: Website = Website::new("https://choosealicense.com");
website.configuration.blacklist_url.push("https://choosealicense.com/licenses/".to_string());
website.configuration.respect_robots_txt = true;
website.configuration.verbose = true;
website.configuration.delay = 2000;
website.crawl();
// ..
~~~

## TODO

- [x] multi-threaded system
- [x] respect _robot.txt_ file
- [x] add configuration object for polite delay, etc..
- [x] add polite delay
- [ ] parse command line arguments

## Contribute

I am open-minded to any contribution. Just fork & `commit` on another branch.