dw-rs 0.2.0

A blazingly fast, parallel download accelerator written in Rust.
Documentation

dw ⚡

dw is a blazingly fast, parallel download accelerator written in Rust.

🚀 Installation

Getting dw is easy. You can choose any of the following methods.

From Crates.io (Recommended)

If you have the Rust toolchain installed, you can install dw directly from the official package registry:

cargo install dw-rs

From Source

If you prefer to build it yourself, you can compile dw directly from the source code:

git clone https://github.com/amirhosseinghanipour/dw.git

cd dw

cargo build --release

The executable will be in ./target/release/dw.

🏁 Getting Started

Using dw is straightforward. Here are the most common commands to get you started.

Basic Download

The simplest way to use dw is to just give it a URL. The tool will automatically figure out the filename and show you a progress bar while it downloads.

dw "http://cachefly.cachefly.net/100mb.test"

Saving with a Different Name

If you want to save the file with a different name, use the -o (or --output) flag.

dw "http://cachefly.cachefly.net/100mb.test" -o "100mb.zip"

🛠️ Advanced Usage

dw exposes a handful of flags to fine-tune the download:

Flag Default Description
-c, --connections <N> 8 Parallel connections. Raise for per-connection-throttled mirrors; lower (or 1) on an already-saturated link.
-o, --output <FILE> from URL Output filename.
-b, --buffer-size <KB> 4096 Write-coalescing size. Larger = fewer disk syscalls.
--piece-size <MB> 4 Work-stealing granularity. Smaller = finer load balancing across uneven connections.
--min-chunk <MB> 1 Minimum file size before parallel download kicks in.
--retries <N> 5 Retry attempts per piece (each resumes from the last written byte).
--timeout <SEC> 60 Per-attempt stall timeout (0 disables).
-q, --quiet off Suppress the progress bar. 
# 16 connections, 2 MiB pieces for finer balancing on a flaky link

dw "http://cachefly.cachefly.net/100mb.test" -c 16 --piece-size 2

⚙️ How It Works

dw is a genuine parallel accelerator, not a wrapper around a single stream:

  • Robust probe. It learns the file size and range support from a HEAD, falling back to a ranged GET (reading Content-Range) when HEAD is uninformative — so it never silently degrades to one connection.
  • Lock-free work stealing. The file is handed out in pieces via a single atomic cursor. Fast connections grab more pieces; the download is never capped by the slowest one.
  • Lock-free progress. Throughput is tracked with one atomic counter sampled off the hot path — no per-packet locking.
  • Buffered positioned writes. Network data is coalesced and written with one positioned write per block on a blocking thread, off the async reactor; the buffer is recycled so steady state allocates nothing.
  • Resilience. Per-piece retries with exponential backoff resume from the last durably-written byte, with a per-attempt stall timeout.

📊 Performance

dw ships a reproducible benchmark (cargo bench) that pits the engine against curl and the previous version over a local throttled server. Highlights on a 4-core host (see BENCHMARKS.md for the full methodology):

  • Scales with connections where it matters: ~5× faster than a single stream at 8 connections on a per-connection-throttled server.
  • Work stealing delivers ~2.6–4× on connections of uneven speed, isolated to the technique alone.
  • Fastest single stream in the suite (≈1.8× curl) thanks to lock-free progress and buffered positioned writes.

An accelerator can't beat physics: on a link whose total bandwidth is the bottleneck (a common home-Internet case), all tools tie and extra connections only add overhead. dw wins when the bottleneck is per-connection — throttling mirrors and high-latency paths.

License

This project is open-source and available under the GPL-3.0 License.