Parallel Downloader (pd) 🦀

A robust, concurrent file downloader built in Rust. It is designed to be resilient, supporting automatic retries, crash recovery, and download verification.

🚀 Features

• Concurrency: Downloads files in parallel chunks to maximize bandwidth.
• Resiliency: Automatically retries failed chunks on network timeouts.
• Crash Recovery: Saves progress to a state file (.state.json). If the program crashes or is interrupted, simply run the command again to resume exactly where it left off.
• Rate Limiting: Optional token-bucket throttling to limit bandwidth usage.
• Integrity Check: Verifies the final file against a SHA-256 hash.
• Library Support: Logic is separated from the CLI, allowing you to use the core downloader in your own Rust applications.
• 🧠 Batch Processing:
  • Supports a Daemon mode (pd start) for background management.
  • Accepts input files (-i list.txt) for bulk downloads.

📦 Installation

Pre-built Binaries

Download the latest release for Windows, Linux, or macOS from the Releases Page.

Build from Source

git clone https://github.com/velumuruganr/parallel_downloader.git
cd parallel_downloader
cargo install --path .

Note: The CLI binary is provided in src/bin/pd/main.rs (the CLI glue is in src/bin/pd/). When developing from source you can run the binary directly with:

# Run the library's CLI binary from source
cargo run --bin pd -- run --url "https://example.com/large.iso"

🛠 Usage

1. Standalone Mode (CLI)

Best for quick, one-off downloads.

# Simple download (defaults to 4 threads)
pd run --url "https://example.com/large.iso"

# Advanced Usage
pd run \
  --url "https://example.com/movie.mp4" \
  --output "holiday.mp4" \
  --threads 8 \
  --rate-limit 1048576 \
  --dir ./downloads

2. Batch Mode

Download a list of URLs (one per line).

# Download files from list.txt, 3 files at a time
pd run -i list.txt -c 3

3. Daemon Mode (Background Service)

Best for long-running servers or managing queues.

# 1. Start the daemon in a separate terminal
pd start

# 2. Add downloads from any terminal
pd add --url "https://example.com/file1.zip"
pd add --url "https://example.com/file2.zip"

# 3. Check status
pd status

# 4. Shutdown
pd stop

Options

Configuration

This project supports reading settings from a config.toml file or from environment variables prefixed with PD.

• File locations (platform-specific):

  • Linux: ~/.config/pd/config.toml
  • macOS: ~/Library/Application Support/pd/config.toml
  • Windows: %APPDATA%\pd\config.toml

• Example: copy config.example.toml to one of the locations above and edit values such as threads, rate_limit, default_dir, and concurrent_files.

• Environment variables: you can override values via environment variables using __ as a separator. For example:

# set threads to 8
export PD__threads=8

# set global rate limit to 1 MiB/s
export PD__rate_limit=1048576

Environment variables take precedence over values found in config.toml.

📚 Library Usage

You can use parallel_downloader as a library in your own project.

Add to your Cargo.toml:

[dependencies]
parallel_downloader = { version = "0.2", default-features = false }

Use the modules in your code:

use parallel_downloader::downloader::prepare_download;
use parallel_downloader::worker::download_chunk;

🧪 Testing

We use wiremock to simulate HTTP failures and chunk ranges without hitting real servers.

cargo test

📂 Examples

The repository includes runnable examples in the examples/ folder to demonstrate how to use parallel_downloader programmatically in your own applications.

Simple Download

This example demonstrates a complete workflow: creating a client, verifying file size, and downloading chunks with a progress bar.

To run the example:

cargo run --example simple_download

You can view the full source code in examples/simple_download.rs.