Crate s3_unspool

Expand description

Streaming ZIP upload and extraction for Amazon S3.

s3-unspool zips local directories and existing S3 prefixes into local or S3 ZIP files, and unzips local or S3 ZIP files into local directories or S3 prefixes.

Local directory and S3-prefix zip operations generate the ZIP once. S3 ZIP destinations are streamed with multipart upload, and local ZIP destinations are written through a temporary sibling file before being renamed into place. Empty local directories and zero-byte S3 marker objects are preserved as ZIP directory entries.

Extraction compares ZIP entries with destination objects listed by ListObjectsV2. Missing objects are uploaded with If-None-Match: *, and changed objects are uploaded with If-Match against the listed destination ETag so newer destination data is not overwritten accidentally. Conditional write conflicts are recorded and skipped by default; set SyncOptions::fail_on_conflict or LocalZipSyncOptions::fail_on_conflict to return an error on the first observed conflict.

Conditional overwrites require s3:GetObject permission on destination objects as well as s3:PutObject. s3-unspool does not issue per-file destination HeadObject requests or read destination object bodies, but S3 authorizes If-Match writes against object-read permission.

Destination PutObject bodies are fed by a source-range scheduler, so a body can pause while waiting for planned ZIP bytes. For high-concurrency extracts, consider relaxing or disabling AWS SDK upload stalled-stream protection on the destination client. Keep download stalled-stream protection enabled for source reads. The repository CLI and Lambda example configure this split. Library users can call sync_zip_to_s3_with_clients when source reads and destination writes need separate S3 client configuration.

inspect_s3_zip reads source ZIP size and file count without downloading the archive. It is useful before choosing memory-sensitive scheduler options with AdaptiveSourceWindow or SyncOptions::with_source_window_memory_budget_mb.

ZIPs created with upload_directory_zip_to_s3, zip_directory_to_file, zip_s3_prefix_to_s3, or zip_s3_prefix_to_file include an embedded catalog at EMBEDDED_CATALOG_PATH by default. The catalog stores each file path and MD5 digest so later extracts can skip unchanged files before decompressing them. Set SyncOptions::force_hash_comparison when you need to measure or force the fallback extract-and-hash path.

Set SyncOptions::with_selection or the equivalent local unzip option when only a subset of ZIP paths should be restored. Selection patterns use gitignore-style syntax and are applied before source range planning, so ranged GetObject requests are planned only for selected entries that still need source bytes. Exclude-only selections restore every non-excluded ZIP path. Selected extracts reject SyncOptions::delete_extra_objects because unselected destination objects are outside the restore scope.

ZIP directory entries round-trip as zero-byte S3 marker objects whose keys end in /. Local upload preserves empty directories as ZIP directory entries, and S3-prefix upload preserves zero-byte S3 marker objects as ZIP directory entries while rejecting nonzero trailing-slash S3 objects as ambiguous.

§Extract an S3 ZIP to a Destination Prefix

use aws_config::BehaviorVersion;
use aws_sdk_s3::Client;
use s3_unspool::{S3Object, S3Prefix, SyncOptions, sync_zip_to_s3};

let config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let client = Client::new(&config);

let extract = SyncOptions::new(
    S3Object::parse("s3://my-bucket/releases/site.zip")?,
    S3Prefix::parse("s3://my-bucket/www/")?,
)
.delete_extra_objects();

let report = sync_zip_to_s3(&client, extract).await?;
println!("uploaded changed files: {}", report.summary.uploaded_changed);

§Extract Selected Entries from an S3 ZIP

use aws_config::BehaviorVersion;
use aws_sdk_s3::Client;
use s3_unspool::{S3Object, S3Prefix, SyncOptions, UnzipSelection, sync_zip_to_s3};

let config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let client = Client::new(&config);

let extract = SyncOptions::new(
    S3Object::parse("s3://my-bucket/releases/site.zip")?,
    S3Prefix::parse("s3://my-bucket/www/")?,
)
.with_selection(
    UnzipSelection::new()
        .include("index.md")
        .include("docs/**/*.md")
        .exclude("docs/drafts/**"),
);

let report = sync_zip_to_s3(&client, extract).await?;
println!("processed entries: {}", report.summary.zip_files);

§Upload a Directory as a Cataloged ZIP

use aws_config::BehaviorVersion;
use aws_sdk_s3::Client;
use s3_unspool::{S3Object, UploadOptions, upload_directory_zip_to_s3};

let config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let client = Client::new(&config);

let upload = UploadOptions::new(
    "./site",
    S3Object::parse("s3://my-bucket/releases/site.zip")?,
);
let report = upload_directory_zip_to_s3(&client, upload).await?;
println!("uploaded files: {}", report.files);

§Upload an S3 Prefix as a Cataloged ZIP

use aws_config::BehaviorVersion;
use aws_sdk_s3::Client;
use s3_unspool::{S3Object, S3Prefix, S3PrefixUploadOptions, zip_s3_prefix_to_s3};

let config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let client = Client::new(&config);

let upload = S3PrefixUploadOptions::new(
    S3Prefix::parse("s3://my-bucket/www/")?,
    S3Object::parse("s3://my-bucket/releases/site.zip")?,
);
let report = zip_s3_prefix_to_s3(&client, upload).await?;
println!("uploaded files: {}", report.files);

§Assumptions

The crate assumes destination objects use single-part S3 ETags that match the MD5 digest of the object body. Multipart destination objects and SSE-C destination ETags are intentionally out of scope for comparison.

Structs§

AdaptiveSourceWindow: Inputs for deriving an adaptive source block window capacity.
DryRunDiagnostics: Effective dry-run source settings and aggregate diagnostics.
DryRunObjectReport: Per-object dry-run operation result.
LocalUnzipDiagnostics: Effective local unzip settings and aggregate source diagnostics.
LocalUnzipOptions: Options for extracting a local ZIP file into a local directory.
LocalUnzipReport: Full report returned when extracting a ZIP into a local directory.
LocalZipOptions: Options for zipping a local directory to a local ZIP file.
LocalZipReport: Summary returned by local ZIP creation helpers.
LocalZipSyncOptions: Options for extracting a local ZIP file into an S3 prefix.
LocalZipToS3Report: Full report returned when extracting a local ZIP into an S3 prefix.
ObjectReport: Per-object operation result.
PutDiagnostics: Destination PutObject failure counters.
PutRetryDiagnostics: Effective destination PutObject retry settings.
PutRetryPolicy: Retry and backoff policy for destination PutObject attempts.
S3Object: Parsed S3 object URI.
S3Prefix: Parsed S3 destination prefix URI.
S3PrefixLocalZipOptions: Options for zipping an S3 prefix to a local ZIP file.
S3PrefixUploadOptions: Options for zipping an S3 prefix and uploading it as an S3 ZIP object.
S3PrefixUploadReport: Summary returned by crate::zip_s3_prefix_to_s3.
S3ZipInfo: Metadata about a source ZIP object stored in S3.
S3ZipLocalUnzipOptions: Options for extracting an S3 ZIP object into a local directory.
SourceDiagnostics: Source scheduler and ranged GetObject counters.
SyncDiagnostics: Effective extract settings and aggregate diagnostics.
SyncOptions: Options for extracting a ZIP object from S3 into an S3 prefix.
SyncReport: Full report returned by crate::sync_zip_to_s3.
SyncSummary: Aggregate counters for an extract run.
UnzipDryRunReport: Full report returned by extract dry-run helpers.
UnzipDryRunSummary: Aggregate counters for an extract dry run.
UnzipSelection: ZIP entry selection patterns for unzip APIs.
UploadOptions: Options for zipping a local directory and uploading it as an S3 object.
UploadProgressHandler: Upload progress callback wrapper.
UploadReport: Summary returned by crate::upload_directory_zip_to_s3.
ZipDryRunReport: Summary returned by ZIP dry-run helpers.

Enums§

ComparisonMode: How unzip operations compare ZIP entries with existing destination objects.
ConflictPolicy: How unzip-to-S3 operations handle destination conditional write conflicts.
DestinationCleanup: How an unzip-to-S3 operation treats destination objects that are not in the ZIP.
DryRunOperationStatus: Status for a single dry-run destination operation.
Error: Error type returned by s3-unspool operations.
OperationStatus: Status for a single destination object operation.
RetryJitter: Jitter mode used for application-level destination PutObject retries.
UploadProgress: Progress event emitted while preparing and streaming an upload ZIP.
ZipCompression: Compression method used for regular file entries when creating ZIP archives.

Constants§

EMBEDDED_CATALOG_PATH: Reserved ZIP entry path used for the optional embedded update catalog.

Functions§

adaptive_source_get_concurrency: Computes adaptive source GetObject concurrency for a fixed memory envelope.
dry_run_sync_zip_to_s3: Plans extracting an S3 ZIP object into an S3 prefix without writing or deleting objects.
dry_run_sync_zip_to_s3_with_clients: Plans extracting an S3 ZIP object into an S3 prefix with separate source and destination clients.
dry_run_unzip_file_to_local: Plans extracting a local ZIP file into a local directory without writing files.
dry_run_unzip_file_to_s3: Plans extracting a local ZIP file into an S3 prefix without writing or deleting objects.
dry_run_unzip_s3_zip_to_local: Plans extracting an S3 ZIP object into a local directory without writing files.
dry_run_upload_directory_zip_to_s3: Plans zipping a local directory and uploading it as an S3 object without writing anything.
dry_run_zip_directory_to_file: Plans zipping a local directory into a local ZIP file without writing anything.
dry_run_zip_s3_prefix_to_file: Plans zipping an S3 prefix into a local ZIP file without writing anything.
dry_run_zip_s3_prefix_to_s3: Plans zipping an S3 prefix and uploading it as an S3 object without writing anything.
inspect_s3_zip: Inspects an S3 ZIP object without downloading it to local storage.
sync_zip_to_s3: Extracts missing or changed files from an S3 ZIP object into an S3 prefix.
sync_zip_to_s3_with_clients: Extracts missing or changed files from an S3 ZIP object into an S3 prefix, using separate S3 clients for source reads and destination writes.
unzip_file_to_local: Extracts a local ZIP file into a local directory.
unzip_file_to_s3: Extracts a local ZIP file into an S3 prefix.
unzip_s3_zip_to_local: Extracts an S3 ZIP object into a local directory.
upload_directory_zip_to_s3: Zips a local directory and uploads the archive to S3.
zip_directory_to_file: Zips a local directory into a local ZIP file.
zip_s3_prefix_to_file: Zips an S3 prefix into a local ZIP file.
zip_s3_prefix_to_s3: Zips an S3 prefix and uploads the archive to S3.

Type Aliases§

Result: Result type returned by s3-unspool operations.

Crate s3_unspool

Crate s3_unspool Copy item path

§Extract an S3 ZIP to a Destination Prefix

§Extract Selected Entries from an S3 ZIP

§Upload a Directory as a Cataloged ZIP

§Upload an S3 Prefix as a Cataloged ZIP

§Assumptions

Structs§

Enums§

Constants§

Functions§

Type Aliases§

Crate s3_unspool