Crate fast_down 

fast-down

fast-down is a fast, concurrent file downloader library built on top of the fast_pull pull/push engine, with first-class HTTP support.

Official Website (Simplified Chinese)

What this crate provides

fast-down re-exports the entire fast_pull pull/push engine (the download_single / download_multi entry points, the Puller and Pusher traits, and the file/memory pushers) and adds the following on top of it:

1. HTTP / reqwest puller — FastDownPuller (with FastDownPullerOptions) is the default Puller for HTTP(S) sources. It builds on a SmartRedirectClient that follows redirects manually so it can honor the Referrer-Policy header and strip resource-specific headers (Origin / Authorization / Cookie) on cross-origin hops, per RFC 9110 §15.4.
2. URL info resolution — UrlInfo and FileId capture a resource’s size, suggested filename, content type, range support, and a stable identity derived from the ETag / Last-Modified headers, which powers incremental and resumable downloads.
3. Proxy support — the Proxy enum selects no proxy, the system proxy, or a custom proxy URL for outgoing requests.
4. Result + event stream — download functions return a DownloadResult whose event_chain receiver disconnects once the pipeline has fully flushed; awaiting that disconnect is the completion signal (no separate join handle).

Supporting building blocks (behind feature flags) include the backend-agnostic http module (HttpClient traits, HttpPuller, Prefetch, ContentDisposition, HttpError) and the reqwest module (SmartRedirectClient, ManualRedirectRequestBuilder).

Example

Download a file concurrently to disk. This requires the reqwest feature (which also enables http and fast-puller) and a network connection, so the block is marked ignore to keep doctests hermetic:

use std::sync::Arc;
use std::time::Duration;
use url::Url;

use fast_down::{FastDownPuller, FastDownPullerOptions, FileId, Proxy};
use fast_pull::StdFilePusher;
use fast_pull::multi::DownloadOptions;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let url = Url::parse("https://example.com/large.bin")?;
    let file = tokio::fs::File::create("large.bin").await?;
    // Pre-size the file; pass `true` to fsync on flush.
    let pusher = StdFilePusher::new(file, /* size */ 0, /* sync_all */ false).await?;

    let puller = FastDownPuller::new(FastDownPullerOptions {
        url,
        headers: Arc::new(Default::default()),
        proxy: Proxy::System,
        accept_invalid_certs: false,
        accept_invalid_hostnames: false,
        cookie_store: false,
        file_id: FileId::default(),
        resp: None,
        available_ips: Arc::from(Vec::<std::net::IpAddr>::new()),
        max_redirects: 10,
    })?;

    let result = fast_pull::download_multi(
        puller,
        pusher,
        DownloadOptions {
            download_chunks: vec![0..u64::MAX].into_iter(),
            concurrent: 8,
            retry_gap: Duration::from_secs(1),
            pull_timeout: Duration::from_secs(30),
            push_queue_cap: 1024,
            min_chunk_size: 1 << 20,
            max_speculative: 3,
        },
    );

    while result.event_chain().recv().await.is_ok() {}
    Ok(())
}

For a sequential, single-threaded download, swap download_multi for fast_pull::download_single and use fast_pull::single::DownloadOptions (which only has retry_gap and push_queue_cap).

Modules

mock

A deterministic in-memory Puller for tests, plus a helper to build mock payloads.

multi

Multi-threaded concurrent download with work-stealing.

single

Single-threaded sequential download.

Structs

BoxPusher

A type-erased pusher that boxes both the pusher and its error type.

BufWriterPusher

Pusher decorator that provides std::io::BufWriter-style linear write buffering.

CacheDirectPusher

Pusher wrapper that buffers chunks and flushes large contiguous runs without merging.

CacheMergePusher

Pusher wrapper that buffers chunks and merges each flush run into a single Bytes.

CacheSeqPusher

Pusher wrapper that reorders out-of-order chunks into sequential order.

DownloadResult

Handle to an active download session.

FileId

File identity used for incremental and resumable downloads.

InvertIter

Iterator that yields the gaps (non-downloaded ranges) from a list of ProgressEntrys.

MemPusher

In-memory pusher for testing or buffer-based workflows.

UrlInfo

Metadata about a downloadable resource, gathered from the initial HTTP request.

Enums

Event

Events emitted during a download session, received via DownloadResult::event_chain.

Proxy

Proxy configuration for outgoing HTTP requests.

Traits

AnyError

Marker trait for type-erased error types.

Merge

Trait for merging a new ProgressEntry into a sorted list of existing entries.

PullStream

A pull stream that yields Bytes chunks.

Puller

Abstraction over a data source that can be pulled (downloaded) in chunks.

PullerError

Extension trait for pull errors, distinguishing recoverable from irrecoverable failures.

Pusher

Abstraction over a data sink that receives pushed byte chunks.

Total

Trait for computing the total size from one or more ProgressEntry values.

Functions

invert

window: when a ProgressEntry length is less than window, it is merged into the gap to reduce progress fragmentation.

Type Aliases

ProgressEntry

A byte-range representing downloaded or to-be-downloaded progress.

ProgressListener

A callback invoked whenever a chunk has been successfully written to its destination (disk / memory).

PullResult

Result type returned by pulling operations.

WorkerId

Numeric identifier assigned to each worker thread/task.