parallel_downloader 0.1.0

A robust, concurrent file downloader designed to be resilient, supporting automatic retries, crash recovery, and download verification.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

# Parallel Downloader 🦀

A robust, concurrent file downloader built in Rust. It is designed to be resilient, supporting automatic retries, crash recovery, and download verification.

## 🚀 Features

* **Concurrency:** Downloads files in parallel chunks to maximize bandwidth.
* **Resiliency:** Automatically retries failed chunks on network timeouts.
* **Crash Recovery:** Saves progress to a state file (`.state.json`). If the program crashes or is interrupted, simply run the command again to resume exactly where it left off.
* **Rate Limiting:** Optional token-bucket throttling to limit bandwidth usage.
* **Integrity Check:** Verifies the final file against a SHA-256 hash.
* **Library Support:** Logic is separated from the CLI, allowing you to use the core downloader in your own Rust applications.

## 📦 Installation

Ensure you have Rust installed. Clone the repository and build:

```bash
git clone https://github.com/velumuruganr/parallel_downloader.git
cd parallel_downloader
cargo build --release
```
The binary will be located in ./target/release/parallel_downloader

## 🛠 Usage

### Basic Download

Download a file with default settings (4 threads).
```bash
cargo run -- --url "https://proof.ovh.net/files/100Mb.dat"
```

### Advanced Usage

Combine flags for a specific use case:
```bash
cargo run -- \
  --url "https://proof.ovh.net/files/100Mb.dat" \
  --output "my_video.dat" \
  --threads 8 \
  --rate-limit 512000 \
  --verify-sha256 "a1b2c3d4..."
```

## Options
| Flag              | Description                                       | Default       |
|-------------------|---------------------------------------------------|---------------|
| `--url`           | The URL to download                               | Required      |
| `--output`, `-o`  | Output filename                                   | `output.bin`  |
| `--threads`, `-t` | Number of concurrent threads                      | `4`           |
| `--rate-limit`    | Max speed in bytes/sec (e.g., 1048576 = 1MB/s)    | Unlimited     |
| `--verify-sha256` | Hash string to verify file integrity              | None          |

## 📚 Library Usage

You can use parallel_downloader as a library in your own project.

Add to your `Cargo.toml`:
```
[dependencies]
parallel_downloader = { path = "../path/to/parallel_downloader" }
```

Use the modules in your code:
```Rust
use parallel_downloader::utils::get_file_size;
use parallel_downloader::worker::download_chunk;

#[tokio::main]
async fn main() {
    let size = get_file_size("https://example.com/file.zip").await.unwrap();
    println!("File size: {}", size);
}
```

## 🧪 Testing
Run the test suite, which includes unit tests for math/hashing and integration tests using a mock HTTP server.
```Bash
cargo test
```

## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.