Crate cobalt_s3_archiver 

Allows ZIP archives to be created in S3 from files stored in S3.

Modules

aws

Provides ways of interacting with objects in S3.

Structs

Archiver

An Archiver allows the creation of a ZIP archive in S3 allowing control of what and how the data is written into S3.

ManifestEntry

A single entry in the manifest file which describes a single entry in the output ZIP archive.

ManifestFileUpload

Serializes a sequence of ManifestEntry instances into S3 as a JSONL file.

ZipEntries

Provide a method of iterating over the details of archived files without reading the entire archive.

Enums

Compression

The compression algorithms that are available to be used to compress entries in the ZIP archive.

Functions

unarchive_all

Extract all files from the ZIP at the zip_file location into the destination prefix. If any of the files in the ZIP are larger than chunk_size then is is uploaded to S3 as a multipart upload.

validate_manifest_file

Validates that all the src files from ManifestEntry records in the manifest file exist, and that their CRC32 values match the value in the manifest. The fetch_concurrency in the number of source files that will be fetched concurrently from S3. The validate_concurrency is the number of source files that will be validated concurrently.

validate_zip_central_dir

Validate that the CRC32 value and filenames in each ManifestEntry in the manifest_file matches those in the central directory of the zip_file.

validate_zip_entry_bytes

Validate each ZIP entry by reading each ManifestEntry from the manifest file and ensuring that the CRC32 of bytes in the ZIP archive match those recorded in the manifest file.