s3-filesystem
This crate is a simple wrapper for file interactions with S3. It enables you to open a file, write a file, and perform a walk through the objects in an S3 bucket. Both reading and writing will create a copy on your machine to make S3 feel as much like your local file system as possible.
This crate utilises the official AWS SDK for S3 operations and uses Tokio for local IO.
When using the crate, be careful! Writing files can overwrite data previously there and using S3 will incur costs.
Usage
First create an OpenOptions struct containing the bucket you wish to connect to and where you wish files to be cached to.
There are then three functions available
- [crate::OpenOptions::open_s3]: Downloads the file and opens it and returns a Tokio File for data to be read from as standard.
- [crate::OpenOptions::write_s3]: Writes the file to disk and to S3, returning the Tokio File.
- [crate::OpenOptions::walkdir]: Walks through the objects in the S3 bucket, with an optional path to walk through a subset of objects.
Open a file
use OpenOptions;
use AsyncReadExt;
async
Write a file
use OpenOptions;
use fs;
const BUCKET: &'static str = "test-bucket";
async
Walkdir
use OpenOptions;
async
Walkdir and download
use OpenOptions;
async
TODOs
- Add feature flags for automatic decompression?
- Look for changes in the file? If bytes is different download, if not read from cache. Beats generic force download config.
Test on more operating systems with more edge cases - currently little testing has occurred.