Module downloader

Available on crate feature downloader only.

Expand description

Handle downloading blobs and collections concurrently and from nodes.

The Downloader interacts with four main components to this end.

ProviderMap: Where the downloader obtains information about nodes that could be used to perform a download.
Store: Where data is stored.

Once a download request is received, the logic is as follows:

The ProviderMap is queried for nodes. From these nodes some are selected prioritizing connected nodes with lower number of active requests. If no useful node is connected, or useful connected nodes have no capacity to perform the request, a connection attempt is started using the DialerT.
The download is queued for processing at a later time. Downloads are not performed right away. Instead, they are initially delayed to allow the node to obtain the data itself, and to wait for the new connection to be established if necessary.
Once a request is ready to be sent after a delay (initial or for a retry), the preferred node is used if available. The request is now considered active.

Concurrency is limited in different ways:

Total number of active request: This is a way to prevent a self DoS by overwhelming our own bandwidth capacity. This is a best effort heuristic since it doesn’t take into account how much data we are actually requesting or receiving.
Total number of connected nodes: Peer connections are kept for a longer time than they are strictly needed since it’s likely they will be useful soon again.
Requests per node: to avoid overwhelming nodes with requests, the number of concurrent requests to a single node is also limited.

Structs§

Getter: Trait modelling performing a single request over a connection. This allows for IO-less testing.
NeedsConn: Trait modelling the intermediary state when a connection is needed to proceed.