fast-down

fast-down is a fast, concurrent file downloader library built on top of the fast_pull pull/push engine, with first-class HTTP support.

Official Website (Simplified Chinese)

What this crate provides

fast-down re-exports the entire fast_pull pull/push engine (the download_single / download_multi entry points, the Puller and Pusher traits, and the file/memory pushers) and adds the following on top of it:

1. HTTP / reqwest puller — FastDownPuller (with FastDownPullerOptions) is the default Puller for HTTP(S) sources. It builds on a SmartRedirectClient that follows redirects manually so it can honor the Referrer-Policy header and strip resource-specific headers (Origin / Authorization / Cookie) on cross-origin hops, per RFC 9110 §15.4.
2. URL info resolution — UrlInfo and FileId capture a resource's size, suggested filename, content type, range support, and a stable identity derived from the ETag / Last-Modified headers, which powers incremental and resumable downloads.
3. Proxy support — the Proxy enum selects no proxy, the system proxy, or a custom proxy URL for outgoing requests.
4. Result + event stream — download functions return a DownloadResult whose event_chain receiver disconnects once the pipeline has fully flushed; awaiting that disconnect is the completion signal (no separate join handle).

Supporting building blocks (behind feature flags) include the backend-agnostic http module (HttpClient traits, HttpPuller, Prefetch, ContentDisposition, HttpError) and the reqwest module (SmartRedirectClient, ManualRedirectRequestBuilder).

Example

Download a file concurrently to disk. This requires the fast-puller feature (which also enables reqwest and http), the file feature, and a network connection, so the block is marked no_run to compile it without making a real network request. The fallback main keeps the example compilable when rustdoc checks a feature combination that does not provide those APIs:

#[cfg(all(feature = "fast-puller", feature = "file"))]
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    use std::sync::Arc;
    use std::time::Duration;
    use url::Url;

    use fast_down::{FileId, Proxy};
    use fast_down::fast_puller::{FastDownPuller, FastDownPullerOptions};
    use fast_pull::StdFilePusher;
    use fast_pull::multi::{DownloadOptions, download_multi};

    let url = Url::parse("https://example.com/large.bin")?;
    let file = tokio::fs::File::create("large.bin").await?;
    // Pre-size the file; pass `true` to fsync on flush.
    let pusher = StdFilePusher::new(file, /* size */ 0, /* sync_all */ false).await?;

    let puller = FastDownPuller::new(FastDownPullerOptions {
        url,
        headers: Arc::new(Default::default()),
        proxy: Proxy::System,
        accept_invalid_certs: false,
        accept_invalid_hostnames: false,
        cookie_store: false,
        file_id: FileId::default(),
        resp: None,
        available_ips: Arc::from(Vec::<std::net::IpAddr>::new()),
        max_redirects: 10,
    })?;

    let result = download_multi(
        puller,
        pusher,
        DownloadOptions {
            download_chunks: vec![0..u64::MAX].into_iter(),
            concurrent: 8,
            retry_gap: Duration::from_secs(1),
            pull_timeout: Duration::from_secs(30),
            push_queue_cap: 1024,
            min_chunk_size: 1 << 20,
            max_speculative: 3,
        },
    );

    while result.event_chain().recv().await.is_ok() {}
    Ok(())
}

#[cfg(not(all(feature = "fast-puller", feature = "file")))]
fn main() {}

For a sequential, single-threaded download, swap download_multi for fast_pull::single::download_single and use fast_pull::single::DownloadOptions (which only has retry_gap and push_queue_cap).