Crate condow_core

Crate condow_core 

Source
Expand description

§ConDow

§Overview

ConDow is a CONcurrent DOWnloader which downloads BLOBs by splitting the download into parts and downloading them concurrently.

Some services/technologies/backends can have their download speed improved, if BLOBs are downloaded concurrently by “opening multiple connections”. An example for this is AWS S3.

This crate provides the core functionality only. To actually use it, use one of the implementation crates:

All that is required to add more “services” is to implement the [CondowClient] trait.

§Usage

To use condow a client to access remote data is required. In the examples below InMemoryClient is used. Usually this would be some client which really accesses remote BLOBs.

The Condow struct itself can be used to download BLOBs. It might not be convenient to pass it around since it has 2 type parameters. Consider the traits Downloads (which has only an associated type) or [DownloadUntyped] (which is even object safe) to pass around instances of Condow.

use condow_core::condow_client::InMemoryClient;
use condow_core::{Condow, config::Config};

// First we need a client...
let client = InMemoryClient::<String>::new_static(b"a remote BLOB");

// ... and a configuration for Condow
let config = Config::default();

let condow = Condow::new(client, config).unwrap();

assert_eq!(condow.get_size("a location").await.unwrap(), 13);

// Download the complete BLOB
let blob = condow.blob().at("a location").download_into_vec().await.unwrap();
assert_eq!(blob, b"a remote BLOB");

// Download part of a BLOB. Any Rust range syntax will work.
let blob = condow.blob().at("a location").range(2..=7).download_into_vec().await.unwrap();
assert_eq!(blob, b"remote");

let blob = condow.blob().at("a location").range(2..).download_into_vec().await.unwrap();
assert_eq!(blob, b"remote BLOB");

// get an `AsyncRead` implementation

use futures::AsyncReadExt;
let mut reader = condow.blob().at("a location").reader().await.unwrap();
let mut buf = Vec::new();
reader.read_to_end(&mut buf).await.unwrap();
assert_eq!(buf, b"a remote BLOB");

// get an `AsyncRead`+`AsyncSeek` implementation
use futures::AsyncSeekExt;
let mut reader = condow.blob()
    .at("a location")
    .trusted_blob_size(13)
    .random_access_reader()
    .finish().await.unwrap();
let mut buf = Vec::new();
reader.seek(std::io::SeekFrom::Start(2)).await.unwrap();
reader.read_to_end(&mut buf).await.unwrap();
assert_eq!(buf, b"remote BLOB");

§Retries

ConDow supports retries. These can be done on the downloads themselves as well on the byte streams returned from a client. If an error occurs while streaming bytes ConDow will try to reconnect with retries and resume streaming where the previous stream failed.

Retries can also be attempted on size requests.

Be aware that some clients might also do retries themselves based on their underlying implementation. In this case you should disable retries for either the client or ConDow itself.

§Behaviour

Downloads with a maximum concurrency of 3 are streamed on the same task the download was initiated on. This means that the returned stream needs to be polled to drive pulling chunks from the network. Executing the streaming also means that panics of underlying libraries will pop up on the polling task.

Downloads with a concurrency greater or equal than 4 are executed on dedicated tasks. Panics will be detected and the stream will abort with an error.

With the EnsureActivePull config setting all downloads will be executed on dedicated tasks and panics will be detected.

All downloads executed on dedicated tasks will pull bytes from the network eagerly and fill a queue.

§Instrumentation

Instrumentation can be done for each individual download or centralized for global monitoring. For further information see the probe module.

Modules§

condow_client
Adapter for crate::Condow to access BLOBs to be downloaded
config
Configuration items
errors
Error types returned by Condow
probe
Probes & Instrumentation
reader
Implementations of async readers for downloads
streams
Stream implememtations used by Condow

Structs§

Condow
The CONcurrent DOWnloader
InclusiveRange
An inclusive range which can not have a length of 0.
OffsetRange
A range defined by an offset and a length.
Request
A request for a download from a specific location
RequestNoLocation
A request for a download where the location is not yet known

Enums§

ClosedRange
A closed range
DownloadRange
A range which specifies a download
OpenRange
An open range

Traits§

Downloads
A common interface for downloading
DownloadsUntyped
Downloads from a location specified by a &str.