Expand description
Transfer Manager for Amazon S3 - a high-performance data plane client for uploading and downloading objects and directories.
The transfer manager splits large transfers into concurrent part requests over the Amazon S3
service API, retries each failed part individually, and sizes its concurrency and memory use to the
host. Client starts transfers; each returns a handle to wait for, control or abort the transfer.
§Examples
Load the default configuration:
let config = aws_s3_transfer::from_env().load().await;
let client = aws_s3_transfer::Client::new(config);Download a bucket to a local directory:
let config = aws_s3_transfer::from_env().load().await;
let client = aws_s3_transfer::Client::new(config);
let handle = client
.download_objects()
.bucket("my-bucket")
.destination("/tmp/my-bucket")
.initiate()?;
// wait for transfer to complete
handle.join().await?;
§Operations
See the documentation for each client operation for more information:
download- download a single objectupload- upload a single objectdownload_objects- download a bucket, or the objects under a prefix set withS3WalkerBuilder::prefix, to a local directoryupload_objects- upload the files in a local directory to a bucket. By default only the directory’s immediate files are uploaded; supplyFsWalker::builder().recursive(true)to upload the whole tree.
§Configuration
from_env returns a ConfigLoader, which loads
S3 client configuration from the shared AWS configuration sources, detects
the host to size the automatic defaults, and applies any settings made on
it. ConfigLoader::s3_config replaces
the shared-configuration load with an existing aws_types::SdkConfig or
S3ClientConfig.
Config::builder constructs a configuration from explicit values
instead. It requires an S3ClientConfig, loads no shared configuration,
and makes no EC2 instance metadata request; the client detects the host
from local system information only. Use it when the S3 configuration is
already complete and neither environment loading nor network detection is
wanted.
RuntimeConfig selects where transfer work runs,
which also determines the HTTP transport used when no HTTP client is
configured; see Runtime and HTTP.
§Integrity
Uploads send the checksums selected by
ChecksumStrategy, which S3
verifies against the bytes it receives. An expected value set with
ChecksumStrategy::with_expected_value
has S3 check the uploaded object against its source.
A completed download reports whether the bytes it delivered were validated
against S3’s checksums:
DownloadOutput::integrity_checks
returns IntegrityChecks, whose
ChecksumValidation is Validated or
says why the bytes were not validated. Validation covers the bytes as
received from S3, not the destination they were written to. A checksum
mismatch fails join() with
ErrorKind::IntegrityError.
Streamed bytes can arrive before the final check, so a download has
succeeded only when join() returns Ok.
A successful download to a path means the operating system accepted every
byte and the file was renamed into place; the transfer manager does not sync
the data to stable storage. The download
module describes when each check applies.
§Runtime and HTTP
By default, transfer work runs on threads the transfer manager creates and
owns (RuntimeConfig::managed), and
requests use the transfer manager’s own HTTP transport, which drives each
connection on the thread that opened it.
RuntimeConfig::multi_thread_tokio
runs transfer work on the caller’s Tokio runtime instead, and requests use
the AWS SDK’s default HTTP client.
An HTTP client set on the aws_types::SdkConfig or S3 configuration is
used with either runtime, in place of the transfer manager’s transport or
the SDK default. To change TLS, DNS or connection settings, configure an
HTTP client. RuntimeConfig
describes what each transport provides.
aws-s3-transfer-ext is an experimental crate that adopts a
managed runtime built by the caller; its API may change in any release.
§Modeled types
S3 request values, response metadata, and enums live in types, and
their builders in types::builders. Operation modules re-export the
modeled types and builders they use.
The sdk-v1 feature enables conversions between modeled values and AWS SDK
for Rust S3 v1 types, conversion from an S3 configuration builder, and
S3WalkContextBuilder construction
from an SDK client.
§WebAssembly
The crate builds for wasm32-wasip1, wasm32-wasip2, and wasm32-wasip3.
Other wasm targets are not supported. On these targets:
- Build with
RUSTFLAGS="--cfg tokio_unstable". The S3 client’s tokio integration enables tokio’sfsfeature, which tokio accepts on wasm only with that flag. - The AWS SDK’s default HTTPS client is not built. Requests fail unless the
aws_types::SdkConfigor S3 configuration supplies an HTTP client, for example one built onwasi:http.wasm32-wasip1has no networking interface for such a client. Managedexecution behaves asRuntimeConfig::multi_thread_tokio, and a current-thread tokio runtime is supported.Client::shutdown_completionreturnsNone. File reads and writes run inline on the thread that polls the transfer.- The
__unstable_extfeature, which adopts a prebuilt managed runtime, is unavailable. - Memory detection is unavailable, so
MemoryBudgetConfig::Autouses its 2 GiB fallback. UseMemoryBudgetConfig::Limitto fit the module’s linear memory.
§Runtime diagnostics
Tracing output and AWS_S3_TM_DIAGNOSTICS are diagnostics, not stable API:
tracing targets, event and span names, their fields, and the settings the
environment variable accepts may change in minor releases. Events are
grouped under these subsystem targets:
aws_s3_transfer::concurrency: concurrency resolution and control.aws_s3_transfer::scheduling: scheduler capacity decisions, worker pool growth, and memory-budget admission.aws_s3_transfer::memory: buffer-pool admission, physical storage, and reclamation.aws_s3_transfer::execution: per-work-item dispatch and completion.aws_s3_transfer::transfer: transfer lifecycle and state transitions.
Other events use the target of the module that emits them.
AWS_S3_TM_DIAGNOSTICS enables optional diagnostic collection without
changing the client configuration API. Settings are namespaced by subsystem
and resolved once when a client configuration is built. A standalone
memory::BufferPool resolves only the memory settings.
The value is a comma-separated list of case-sensitive key=value settings:
AWS_S3_TM_DIAGNOSTICS=memory.snapshot=1000ms,memory.detail=1,transfer.detail=1Whitespace around entries, keys, and values is ignored. Later valid assignments replace earlier ones. Unknown keys are ignored for forward compatibility. A malformed recognized setting produces a warning and keeps the preceding value. Unsupported detail levels use the highest level understood by this version and produce a warning.
The environment variable controls collection cost. Tracing filters independently control whether collected records are emitted.
§Memory pool
Memory diagnostics report pool capacity, admission pressure, allocation fallback, preparation, and reclamation.
memory.snapshot=offdisables periodic reports. A positive integer followed bymsenables reports; values below100msuse100ms.memory.detail=0uses counters updated only at pressure and lifecycle boundaries.memory.detail=1also counts every optimistic allocation attempt and bitmap word inspected, adding relaxed atomic updates to the acquisition path.
Memory snapshots use the aws_s3_transfer::memory tracing target
at DEBUG:
AWS_S3_TM_DIAGNOSTICS=memory.snapshot=1000ms
RUST_LOG=aws_s3_transfer::memory=debugSnapshot reporting reuses the memory pool’s maintenance thread. It does not create a diagnostics-only thread. Detailed counters and periodic reporting are independent; both are disabled by default.
§Transfer state machines
Transfer diagnostics report where an operation spends time and how work moves through its direction-specific state machine.
transfer.detail=0disables optional transfer-state collection.transfer.detail=1collects an aggregate summary for each upload, download,upload_objectsanddownload_objectstransfer.transfer.detail=2also collects individual state transitions.
Transfer summaries use the operation’s tracing target at DEBUG;
level-2 transitions use the same target at TRACE. Each operation logs
under its own target below aws_s3_transfer::transfer, so the
transfer target enables all of them:
AWS_S3_TM_DIAGNOSTICS=transfer.detail=1
RUST_LOG=aws_s3_transfer::transfer=debugRe-exports§
Modules§
- client
- The transfer manager client and its operations.
- config
- Client configuration and loading: S3 settings, part size, concurrency, read-ahead, runtime and memory.
- error
- Errors returned by transfers, and how to classify them.
- io
- Upload bodies, custom part streams, and walkers for local directories and S3 buckets.
- memory
- Payload-memory configuration and shared pooled storage.
- metrics
- Transfer progress and I/O metrics, and client memory metrics.
- operation
- Inputs, builders, handles and outputs for each operation.
- types
- S3 values and response metadata, transfer policies and outcomes, and byte-size and throughput units.
Structs§
- Scheduling
Ctl - Scheduling controls for a transfer.
- Shutdown
Completion - Observes managed-runtime completion without retaining the client or owner.
- Shutdown
Error - Failure to observe transfer-manager-owned runtime completion.
Enums§
- Shutdown
Error Kind - Classifies a failure to observe managed-runtime shutdown.
Functions§
- from_
env - Create a
ConfigLoaderthat loads configuration from the environment.