Crate aws_multipart_upload

Expand description

§aws-multipart-upload

A high-level API for building and working with AWS S3 multipart uploads using the official SDK for Rust.

§Overview

As explained in the README, the goal of this crate is to provide an API that simplifies the process of performing S3 multipart uploads with abstractions that hide the tedious and precise details, and in a way that is easily compatible with the more ubiquitous dependencies from the ecosystem.

The crate exports several types that implement the trait MultipartWrite, each being an aspect of the multipart upload:

A buffer for polling part upload request futures.
A type that creates part upload request objects, pushes them to such a buffer, and completes the upload when requested.
An interface for encoding arbitrary values in the body of a part upload request.

Combined with any SendRequest, these components are collected in the type MultipartUpload, which is able to manage the end-to-end lifecycle of a single multipart upload, or a series of them continuing indefinitely. The capability is generally captured in the trait alias/abbreviation AwsMultipartUpload.

Combinators from the multipart-write crate can be used to chain and compose types here. The extension traits UploadWriteExt and UploadStreamExt expand on this to allow creating general AwsMultipartUploads, futures for running uploads, and streams as an additional context for a multipart upload.

§Example

Generally, start with an UploadBuilder, which is used to configure several things:

Constraints on the uploaded object itself: target size, part size, etc.
A PartEncoder that specifies how values are written to parts.
Some SendRequest, most likely an SdkClient.

Building this returns a MultipartUpload that can be used in a number of different ways. The following example shows how a MultipartUpload can be used more manually, in that the upload happens with explicit method calls. More examples can be found in the README or repository.

use aws_multipart_upload::{ByteSize, SdkClient, UploadBuilder};
use aws_multipart_upload::codec::JsonLinesEncoder;
use multipart_write::MultipartWriteExt as _;
use serde_json::{Value, json};

// Build a default multipart upload client from `aws_sdk_s3::Client`.
//
// For convenience `aws_config` is re-exported, as is `aws_sdk_s3` under the
// symbol `aws_sdk`, for customization.
let client = SdkClient::defaults().await;

// Use `UploadBuilder` to make a multipart upload with target size 20 MiB,
// target part size 5 MiB, and which writes incoming `serde_json::Value`s
// to parts as jsonlines.
let mut upl = UploadBuilder::new(client)
    .upload_size(ByteSize::mib(20))
    .part_size(ByteSize::mib(5))
    .with_encoder(JsonLinesEncoder)
    .with_uri(("a-bucket-us-east-1", "an/object/key.jsonl"))
    .build();

// Now the uploader can have `serde_json::Value`s written to it to build a
// part of the upload.
//
// As parts reach the target size of 5 MiB, they'll be turned into a request
// body for a part upload and the request will be sent.
for n in 0..100000 {
    let item = json!({"k1": n, "k2": n.to_string()});
    let status = upl.send_part(item).await?;
    println!("bytes written to part: {}", status.part_bytes);

    // We've reached target upload size:
    if status.should_upload {
        let res = upl.complete().await?;
        println!("created {} with entity tag {}", res.uri, res.etag);
        break;
    }
}

Modules§

codec: Encoding data in the body of a part upload request.
error: Types for working with errors.
request: Request interface of the multipart upload API.
uri: ObjectUri iterators.
write: A collection of MultipartWrite implementations for multipart uploads.

Structs§

ByteSize: Byte size representation.
ObjectUri: The address of an uploaded object in S3.
ObjectUriIter: Produce an ObjectUri for a new upload from an iterator.
SdkClient: AWS S3 SDK client.
Status: Value returned by the EncodedUpload writer.
UploadBuilder: Configures and builds a type for multipart uploads.
UploadClient: A client of the multipart upload API.

Traits§

AwsMultipartUpload: Trait alias for a general form of MultipartUpload.
SendRequest: SendRequest represents the atomic operations in a multipart upload.

Type Aliases§

MultipartUpload: A type for creating, building, and completing a multipart upload.