spider 1.0.1

Web spider framework that can spider a domain and collect pages it visits.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

# Spider

Web spider framework that can spider a domain and collect pages it visits.

## Depensencies

~~~bash
$ apt install openssl libssl-dev
~~~

## Usage

## from source for command line usages

~~~bash
$ git clone https://github.com/madeindjs/spider.git
$ cd spider
$ cargo build --release
$ ./target/debug/rust-crawler http://localhost:4000
~~~

Will produce something like this

    - http://localhost:4000/
    - http://localhost:4000/portfolio
    - http://localhost:4000/resume
    - http://localhost:4000/blog

## as crate for librairy usage

Add this dependency to your _Cargo.toml_ file.

~~~toml
[dependencies]
TODO
~~~

## Use

~~~rust
// insert crate here

let mut localhost = Website::new("http://localhost:4000");
localhost.crawl();

for page in localhost.get_pages() {
    println!("- {}", page.get_url());
}

// TODO
~~~


## TODO

- [ ]: multi-threaded system
- [ ]: respect _robot.txt_ file
- [ ]: add configuratioon object for polite delay, etc..
- [ ]: parse command line arguments

## Contribute

I am open-minded to any contribution. Just fork & `commit` on another branch.