maman 0.4.0

Rust Web Crawler
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

# Maman

Maman is a Rust Web Crawler saving pages on Redis.

Pages are send to list `<MAMAN_ENV>:queue:maman` using
[Sidekiq](https://github.com/mperham/sidekiq/wiki/Job-Format) job format:

``` json
{
"class": "Maman",
"jid": "b4a577edbccf1d805744efa9",
"retry": true,
"created_at": 1461789979, "enqueued_at": 1461789979,
"args": {
    "document":"<html><body><a href='#' /><a href='/new' /></html>",
    "urls": ["http://example.net/new"],
    "extra": [],
    "headers": {"content-type": "text/html"},
    "url": "http://example.net/"
    }
}
```

## Dependencies

* Redis

## Installation

~~~
cargo install maman
~~~

## Usage

~~~
REDIS_URL="redis://127.0.0.1/" maman URL
~~~

## LICENSE

The MIT License

Copyright (c) 2016 Laurent Arnoud <laurent@spkdev.net>

---
[![Build](https://img.shields.io/travis-ci/spk/maman.svg)](https://travis-ci.org/spk/maman)
[![Version](https://img.shields.io/crates/v/maman.svg)](https://crates.io/crates/maman)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](http://opensource.org/licenses/MIT "MIT")
[![Project status](http://img.shields.io/status/experimental.png?color=red)](https://github.com/spk/maman)