Struct s3_filesystem::OpenOptions

pub struct OpenOptions { /* private fields */ }

Holds configuration data for syncing S3 objects.

Bucket will specify the bucket which is mounted at mount_path. It will download the file from the bucket to the path maintaining the same folder structure. If force_download is set to true, it will always download the files from S3. If it’s false, it will use whatever is found on disk at that location.

Implementations

impl OpenOptions

pub async fn new(bucket: String, client: Option<Client>) -> Self

Create a new OpenOptions struct.

This function should be used to create a new option configuration for your S3 bucket and filesystem. If data is needed from another bucket, a new OpenOptions should be created.

Client is an optional argument - if it exists that will be the client used and if it doesn’t, this function will automatically create an S3 client from your environment (the AWS CLI).

If non default mount paths are wanted, the function OpenOptions::mount_path can be used, and if you wish to re-download data each time, OpenOptions::force_download can be used.

Examples

 use s3_filesystem::OpenOptions;

 #[tokio::main]
 async fn main() {
  let open_options = OpenOptions::new(bucket, None)
     .await
     .mount_path("data/test/")
     .force_download(true);
 }

pub fn mount_path<P>(self, folder_path: P) -> Self

where P: Into<PathBuf>,

Attach a custom mount path.

By default any data downloaded from S3 is found in target/temp. This can be changed by using this function!

pub fn force_download(self, download: bool) -> Self

Enforce downloading the data every time

Cache is supported by default - if a file with the same name is found on disk then it is read in. Pass download = true if you wish to disable this behavior.

impl OpenOptions

pub async fn open_s3<P>( &self, path: P ) -> Result<File, S3FilesystemError<GetObjectError, HttpResponse>>

where P: AsRef<Path>,

Open a file from S3.

This function will find the S3 file, download it and return a tokio::fs::File ready to be read. Doing it this way enables large files to be downloaded in chunks as well as local caching..

Files will be placed in the mount_path and all folder structure is retained. Folders will be created if they do not exist already.

Arguments

• path: The path, including filename, of the file to be downloaded and opened.

 use s3_filesystem::OpenOptions;
 use tokio::io::AsyncReadExt;

 #[tokio::main]
 async fn main() {
  let open_options = OpenOptions::new(bucket, None)
     .await
     .mount_path("data/test/")
     .force_download(true);

 let mut file = open_options
     .open_s3("redasa1-Q1-20/manifest.txt")
     .await
     .unwrap();

  let mut string = String::new();

  file.read_to_string(&mut string).await.unwrap();

  println!("String: {}", string);
 }

pub async fn write_s3<P>( &self, path: P, buf: &[u8] ) -> Result<File, S3FilesystemError<PutObjectError, HttpResponse>>

where P: AsRef<Path>,

Write a file to S3

Enter a path relative to the bucket and this function will create a file in S3 and on your local system under the mount path chosen in OpenOptions. This will overwrite any files that exist with the same name and will return the file that has been written to.

Arguments

• path: The path, including the filename, where you wish to store the data.
• buf: The data you wish to store.

Examples

use s3_filesystem::OpenOptions;
use tokio::fs;

const BUCKET: &'static str = "test-bucket";

#[tokio::main]
async fn main() {
    let bucket = BUCKET.to_string();

    let open_options = OpenOptions::new(bucket, None)
        .await
        .mount_path("data/test/")
        .force_download(true);

    let data = fs::read("data/manifest.txt").await.unwrap();

    open_options.write_s3("manifest.txt", &data).await.unwrap();

    println!("Data uploaded successfully");
}

pub async fn walkdir<P>( &self, path: P ) -> Result<Vec<DirEntry>, S3FilesystemError<ListObjectsV2Error, HttpResponse>>

where P: AsRef<Path>,

Return a list of S3 objects within the bucket

This function returns the files and folders (S3 objects) in the bucket defined in OpenOptions. A sub path can be specified to return a subset of the items - for the entire bucket provide an empty string: “”.

It returns their path, size, and whether or not it’s a directory, but be wary - directories do not exist in S3. This function will return any directories that have been created as a dummy object ending in “/” within S3. It is not guaranteed to find all directories. This may change in upcoming versions.

Arguments

• path: A path to search within the S3 bucket. If you want the entire bucket, just specify an empty string: “”.

Examples

use s3_filesystem::OpenOptions;

#[tokio::main]
async fn main() {
    let bucket = "my_aws_s3_bucket".to_string();

    let open_options = OpenOptions::new(bucket, None).await;

    let data = open_options.walkdir("").await.unwrap();

    for dat in data {
        println!("Data: {:?}", dat);
    }
}

Trait Implementations

impl Clone for OpenOptions

fn clone(&self) -> OpenOptions

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for OpenOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.