Transfer Manager for Amazon S3

Transfer Manager for Amazon S3 - a high-performance data plane client for uploading and downloading objects and directories.
Key features and benefits
- Automatic Request Splitting: Improves throughput by automatically splitting large requests into part-sized chunks and processing them in parallel across multiple connections. This overcomes single-connection bandwidth limitations.
- Automatic Retries: Enhances reliability by automatically retrying failed chunks independently, eliminating the need to restart entire transfers when intermittent errors occur.
- Optimized Parallel Reads: Implements parallel read operations across different sections of large files during uploads, providing superior performance compared to sequential reading.
- Simplified Directory Operations: Upload the files in a local directory, optionally recursing into its subdirectories, or download the objects sharing a common prefix with a single API call, streamlining bulk transfer operations.
- Automatic sizing: part size, concurrency and memory use are sized to the host by default, and each can be set explicitly.
Getting started
Add the crate to your project:
Create a client:
from_env loads the S3 client configuration from the shared AWS configuration sources and sizes the automatic defaults to the host.
use Client;
async
Upload a file: A large file is split into part-sized chunks that are uploaded in parallel.
use InputStream;
use Client;
async
Download to a file: A large object is downloaded in parallel part-sized ranges. The data is written to a temporary file in the destination's directory, which is renamed to the destination when the download succeeds.
use Client;
async
Stream a download:
The body delivers the object's data in order as its parts arrive. join returns the result of the transfer, including any checksum mismatch; accept the data only after it succeeds.
use Client;
async
Upload a directory:
By default only the files directly inside the source directory are uploaded. This example supplies a recursive walker, so it uploads every file in the directory tree, combining the given prefix with each file's path from the filesystem.
For example, if your prefix is "prefix" and the file path is "test/docs/key.json", it will be uploaded with the key "prefix/test/docs/key.json".
use FsWalker;
use Client;
async
Download a prefix:
This will download every object under the prefix, writing each one at its full key under the destination directory. With the prefix "photos/", the object "photos/2023/a.jpg" is written to "/path/to/directory/photos/2023/a.jpg".
use S3Walker;
use Client;
async
Learn more
The API documentation on docs.rs describes the client configuration, each transfer operation and its output, integrity checks, runtime and HTTP options, and runtime diagnostics.
Getting help
- GitHub discussions - For ideas, RFCs & general questions
- GitHub issues - For bug reports & feature requests
Contributing
See CONTRIBUTING.
License
This project is licensed under the Apache-2.0 License.