Crate fast_down 

fast-down

fast-down is a fast, concurrent file downloader library built on top of the fast_pull pull/push engine, with first-class HTTP support.

Official Website (Simplified Chinese)

What this crate provides

fast-down re-exports the entire fast_pull pull/push engine (the download_single / download_multi entry points, the Puller and Pusher traits, and the file/memory pushers) and adds the following on top of it:

1. HTTP / reqwest puller — FastDownPuller (with FastDownPullerOptions) is the default Puller for HTTP(S) sources. It builds on a SmartRedirectClient that follows redirects manually so it can honor the Referrer-Policy header and strip resource-specific headers (Origin / Authorization / Cookie) on cross-origin hops, per RFC 9110 §15.4.
2. URL info resolution — UrlInfo and FileId capture a resource’s size, suggested filename, content type, range support, and a stable identity derived from the ETag / Last-Modified headers, which powers incremental and resumable downloads.
3. Proxy support — the Proxy enum selects no proxy, the system proxy, or a custom proxy URL for outgoing requests.
4. Result + event stream — download functions return a DownloadResult whose event_chain receiver disconnects once the pipeline has fully flushed; awaiting that disconnect is the completion signal (no separate join handle).

Supporting building blocks (behind feature flags) include the backend-agnostic http module (HttpClient traits, HttpPuller, Prefetch, ContentDisposition, HttpError) and the reqwest module (SmartRedirectClient, ManualRedirectRequestBuilder).

Example

Download a file concurrently to disk. This requires the fast-puller feature (which also enables reqwest and http), the file feature, and a network connection, so the block is marked no_run to compile it without making a real network request. The fallback main keeps the example compilable when rustdoc checks a feature combination that does not provide those APIs:

#[cfg(all(feature = "fast-puller", feature = "file"))]
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    use std::sync::Arc;
    use std::time::Duration;
    use url::Url;

    use fast_down::{FileId, Proxy};
    use fast_down::fast_puller::{FastDownPuller, FastDownPullerOptions};
    use fast_pull::StdFilePusher;
    use fast_pull::multi::{DownloadOptions, download_multi};

    let url = Url::parse("https://example.com/large.bin")?;
    let file = tokio::fs::File::create("large.bin").await?;
    // Pre-size the file; pass `true` to fsync on flush.
    let pusher = StdFilePusher::new(file, /* size */ 0, /* sync_all */ false).await?;

    let puller = FastDownPuller::new(FastDownPullerOptions {
        url,
        headers: Arc::new(Default::default()),
        proxy: Proxy::System,
        accept_invalid_certs: false,
        accept_invalid_hostnames: false,
        cookie_store: false,
        file_id: FileId::default(),
        resp: None,
        available_ips: Arc::from(Vec::<std::net::IpAddr>::new()),
        max_redirects: 10,
    })?;

    let result = download_multi(
        puller,
        pusher,
        DownloadOptions {
            download_chunks: vec![0..u64::MAX].into_iter(),
            concurrent: 8,
            retry_gap: Duration::from_secs(1),
            pull_timeout: Duration::from_secs(30),
            push_queue_cap: 1024,
            min_chunk_size: 1 << 20,
            max_speculative: 3,
        },
    );

    while result.recv().await.is_ok() {}
    Ok(())
}

#[cfg(not(all(feature = "fast-puller", feature = "file")))]
fn main() {}

For a sequential, single-threaded download, swap download_multi for fast_pull::single::download_single and use fast_pull::single::DownloadOptions (which only has retry_gap and push_queue_cap).

Modules

fast_puller

The default HTTP fast_pull::Puller for this crate.

getifaddrs

Enumeration of usable local network interfaces.

http

A backend-agnostic HTTP download layer.

mock

A deterministic in-memory Puller for tests, plus a helper to build mock payloads.

multi

Multi-threaded concurrent download with work-stealing.

reqwest

A reqwest-based implementation of the crate::http HTTP traits with smart redirect handling.

single

Single-threaded sequential download.

Structs

BoxPusher

A type-erased pusher that boxes both the pusher and its error type.

BufWriterPusher

Pusher decorator that provides std::io::BufWriter-style linear write buffering.

CacheDirectPusher

Pusher wrapper that buffers chunks and flushes large contiguous runs without merging.

CacheFilePusher

File pusher combining CacheSeqPusher with StdFilePusher.

CacheMergePusher

Pusher wrapper that buffers chunks and merges each flush run into a single Bytes.

CacheSeqPusher

Pusher wrapper that reorders out-of-order chunks into sequential order.

DownloadResult

Handle to an active download session.

FileId

File identity used for incremental and resumable downloads.

InvertIter

Iterator that yields the gaps (non-downloaded ranges) from a list of ProgressEntrys.

MemPusher

In-memory pusher for testing or buffer-based workflows.

MmapFilePusher

File pusher using memory-mapped I/O for zero-copy writes.

StdFilePusher

File pusher using a raw std::fs::File with random-access writes and no userspace write buffer.

UrlInfo

Metadata about a downloadable resource, gathered from the initial HTTP request.

Enums

Event

Events emitted during a download session, received via DownloadResult::event_chain.

Proxy

Proxy configuration for outgoing HTTP requests.

Traits

AnyError

Marker trait for type-erased error types.

Merge

Trait for merging a new ProgressEntry into a sorted list of existing entries.

PullStream

A pull stream that yields Bytes chunks.

Puller

Abstraction over a data source that can be pulled (downloaded) in chunks.

PullerError

Extension trait for pull errors, distinguishing recoverable from irrecoverable failures.

Pusher

Abstraction over a data sink that receives pushed byte chunks.

Total

Trait for computing the total size from one or more ProgressEntry values.

Functions

invert

window: when a ProgressEntry length is less than window, it is merged into the gap to reduce progress fragmentation.

Type Aliases

ProgressEntry

A byte-range representing downloaded or to-be-downloaded progress.

ProgressListener

A callback invoked whenever a chunk has been successfully written to its destination (disk / memory).

PullResult

Result type returned by pulling operations.

WorkerId

Numeric identifier assigned to each worker thread/task.