stream_crawler 0.1.1

A crate for scraping web pages and extracting URLs and endpoints.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106

# Stream crawler

`stream-scraper` is a Rust crate that provides an asynchronous web crawling utility. It processes URLs, extracts content and child URLs, and handles retry attempts for failed requests. It uses the `tokio` runtime for asynchronous operations and the `reqwest` library for HTTP requests.

## Features

- Asynchronous crawling using `tokio`
- Extracts URLs from `<a>` tags in HTML
- Retries failed requests up to a specified number of attempts
- Limits the number of concurrent requests using a semaphore

## Installation

Add this to your `Cargo.toml`:

```toml
[dependencies]
stream_crawler = "0.1.0"
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.11", features = ["json"] }
scraper = "0.12"
```

## Usage

```rust
use stream_crawler::scrape;
use tokio_stream::StreamExt;

#[tokio::main]
async fn main() {
    let urls = vec![
        String::from("https://www.google.com"),
        String::from("https://www.twitter.com"),
    ];

    let mut result_stream = scrape(urls, 3, 5, 10).await;

    while let Some(data) = result_stream.next().await {
        println!("Processed URL: {:?}", data);
    }
}

```

### Functionality

1. **`scrape` function** :

* Takes a vector of URLs, a retry attempt limit, and a maximum number of concurrent processes.
* Returns a stream of `ProcessedUrl` structures.

1. **`ProcessedUrl` structure** :

* Contains the original URL, the parent URL (if any), the HTML content of the page, and a list of child URLs extracted from `<a>` tags.

### Example

This example demonstrates how to use the `scrape` function to process a list of URLs.

```rust
use stream_crawler::scrape;
use tokio_stream::StreamExt;

#[tokio::main]
async fn main() {
    let urls = vec![
        String::from("https://www.google.com"),
        String::from("https://www.twitter.com"),
    ];

    let mut result_stream = scrape(urls, 3, 5, 10).await;

    while let Some(data) = result_stream.next().await {
        println!("Processed URL: {:?}", data);
    }
}

```

## Documentation

Refer to the inline documentation for detailed usage and examples.

### ` ProcessedUrl`

```rust
#[derive(Debug, PartialEq)]
pub struct ProcessedUrl {
    pub parent: Option<String>,
    pub url: String,
    pub content: String,
    pub children: Vec<String>,
}

```

## Contributing

Contributions are welcome! Please open an issue or submit a pull request.

## License

This project is licensed under the MIT License.

This `README.md` provides an overview of the crate, its features, installation instructions, and usage examples. You can customize it further based on your specific requirements.