crawly 0.1.9

A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

# Contributing to crawly 🕷️

First off, thank you for considering contributing to `crawly`! It's people like you that make `crawly` such a great
tool.

## Code of Conduct

Every community member is expected to follow our [Code of Conduct](CODE_OF_CONDUCT.md). Please make sure you are
welcoming and respectful to everyone.

## Where can I ask for help?

If you have any questions, feel free to ask in the [issues section](https://github.com/aichat-bot/crawly/issues). Label
your issue as `question`.

## How to contribute

1. **Fork the repository**: Start by forking the `crawly` repository.

2. **Clone the forked repository**:

```bash
git clone https://github.com/YOUR-USERNAME/crawly.git
cd crawly
```

3. **Create a new branch**:

```bash
git checkout -b my-new-feature
Make Changes: Add your feature or fix bugs and make sure to commit your changes.
```

4. **Push to GitHub**:

```bash
git push origin my-new-feature
```

5. **Submit a Pull Request**: Go to the crawly GitHub page and click on "New Pull Request". Select your fork and the
   branch you created. Click "Create Pull Request".

6. **Address Feedback**: Sometimes, your changes might not be accepted as is. Typically, a maintainer will review and
   ask for any necessary changes before they merge it.

## 🐛 Reporting Bugs

1. **Check Existing Issues**: Ensure that the bug was not already reported by searching on GitHub
   under [Issues](https://github.com/aichat-bot/crawly/issues).

2. **Provide Information**: If the bug hasn't been reported, open a new issue. Fill out the bug report with details
   like:
    - **Description of the problem**: A clear and concise description of what the bug is.
    - **Steps to reproduce**: Provide step-by-step sequence to reproduce the issue.
    - **Expected behavior**: Describe what you expected to happen.
    - **Actual behavior**: What actually happened.
    - **Screenshots**: If applicable, add screenshots to help explain the problem.
    - **Environment details**: Mention your OS, Rust version, and any other relevant details.

## ✨ Suggesting Enhancements

1. **Check Existing Issues**: Ensure that the enhancement hasn't been suggested by searching
   the [issues list](https://github.com/aichat-bot/crawly/issues).

2. **Provide Information**: If you have a new idea, open a new issue. Detail the enhancement with:
    - **A clear and descriptive title**: This helps others understand the suggestion.
    - **Detailed description**: Explain the core ideas of the enhancement.
    - **Reasons**: Why do you believe this enhancement would be beneficial.
    - **Use case scenarios**: Describe how this enhancement might be used by end-users.

## 🛠️ Setting up your development environment

1. **Install Rust**: If you haven't already, install Rust by following the
   instructions [here](https://www.rust-lang.org/learn/get-started).

2. **Fork and clone**: Follow the steps mentioned in the "How to contribute" section.

3. **Build**: Run `cargo build` to ensure everything is set up correctly.

4. **Run tests**: Before pushing any changes, ensure you run and pass all tests with `cargo test`.

Remember, contributing isn't just about code! You can help by improving documentation, designing, and more. Every
contribution is appreciated.