Crate condow_core

Expand description

ConDow

Overview

ConDow is a CONcurrent DOWnloader which downloads BLOBs by splitting the download into parts and downloading them concurrently.

Some services/technologies/backends can have their download speed improved, if BLOBs are downloaded concurrently by “opening multiple connections”. An example for this is AWS S3.

This crate provides the core functionality only. To actually use it, use one of the implementation crates:

condow_rusoto for downloading AWS S3 via the rusoto
condow_fs for using async file access via tokio

All that is required to add more “services” is to implement the CondowClient trait.

Usage

To use condow a client to access remote data is required. In the examples below InMemoryClient is used. Usually this would be some client which really accesses remote BLOBs.

The Condow struct itself can be used to download BLOBs. It might not be convinient to pass it around since it has 2 type parameters. Consider the traits Downloads (which has only an associated type) or [DownloadUntyped] (which is even object safe) to pass around instances of Condow.

use condow_core::condow_client::InMemoryClient;
use condow_core::{Condow, config::Config};

// First we need a client...
let client = InMemoryClient::<String>::new_static(b"a remote BLOB");

// ... and a configuration for Condow
let config = Config::default();

let condow = Condow::new(client, config).unwrap();

assert_eq!(condow.get_size("a location").await.unwrap(), 13);

// Download the complete BLOB
let blob = condow.blob().at("a location").download_into_vec().await.unwrap();
assert_eq!(blob, b"a remote BLOB");

// Download part of a BLOB. Any Rust range syntax will work.
let blob = condow.blob().at("a location").range(2..=7).download_into_vec().await.unwrap();
assert_eq!(blob, b"remote");

let blob = condow.blob().at("a location").range(2..).download_into_vec().await.unwrap();
assert_eq!(blob, b"remote BLOB");

Retries

ConDow supports retries. These can be done on the downloads themselves as well on the byte streams returned from a client. If an error occurs while streaming bytes ConDow will try to reconnect with retries and resume streaming where the previous stream failed.

Retries can also be attempted on size requests.

Be aware that some clients might also do retries themselves based on their underlying implementation. In this case you should disable retries for either the client or ConDow itself.