Struct ObjectStore 

pub struct ObjectStore {
    pub inner: Arc<dyn ObjectStore>,
    pub use_constant_size_upload_parts: bool,
    pub list_is_lexically_ordered: bool,
    pub store_prefix: String,
    /* private fields */
}

Wraps ObjectStore

Fields

inner: Arc<dyn ObjectStore>

use_constant_size_upload_parts: bool

Whether to use constant size upload parts for multipart uploads. This is only necessary for Cloudflare R2.

list_is_lexically_ordered: bool

Whether we can assume that the list of files is lexically ordered. This is true for object stores, but not for local filesystems.

store_prefix: String

The datastore prefix that uniquely identifies this object store. It encodes information which usually cannot be found in the URL such as Azure account name. The prefix plus the path uniquely identifies any object inside the store.

Implementations

impl ObjectStore

pub async fn from_uri(uri: &str) -> Result<(Arc<ObjectStore>, Path), Error>

Parse from a string URI.

Returns the ObjectStore instance and the absolute path to the object.

This uses the default ObjectStoreRegistry to find the object store. To allow for potential re-use of object store instances, it’s recommended to create a shared ObjectStoreRegistry and pass that to Self::from_uri_and_params.

pub async fn from_uri_and_params( registry: Arc<ObjectStoreRegistry>, uri: &str, params: &ObjectStoreParams, ) -> Result<(Arc<ObjectStore>, Path), Error>

Parse from a string URI.

Returns the ObjectStore instance and the absolute path to the object.

pub fn extract_path_from_uri( registry: Arc<ObjectStoreRegistry>, uri: &str, ) -> Result<Path, Error>

Extract the path component from a URI without initializing the object store.

This is a synchronous operation that only parses the URI and extracts the path, without creating or initializing any object store instance.

Arguments

• registry - The object store registry to get the provider
• uri - The URI to extract the path from

Returns

The extracted path component

pub fn from_path(str_path: &str) -> Result<(Arc<ObjectStore>, Path), Error>

👎Deprecated:

Use from_uri instead

pub fn local() -> ObjectStore

Local object store.

pub fn memory() -> ObjectStore

Create a in-memory object store directly for testing.

pub fn is_local(&self) -> bool

Returns true if the object store pointed to a local file system.

pub fn is_cloud(&self) -> bool

pub fn scheme(&self) -> &str

pub fn block_size(&self) -> usize

pub fn max_iop_size(&self) -> u64

pub fn io_parallelism(&self) -> usize

pub fn io_tracker(&self) -> &IOTracker

Get the IO tracker for this object store

The IO tracker can be used to get statistics about read/write operations performed on this object store.

pub fn io_stats_snapshot(&self) -> IoStats

Get a snapshot of current IO statistics without resetting counters

Returns the current IO statistics without modifying the internal state. Use this when you need to check stats without resetting them.

pub fn io_stats_incremental(&self) -> IoStats

Get incremental IO statistics since the last call to this method

Returns the accumulated statistics since the last call and resets the counters to zero. This is useful for tracking IO operations between different stages of processing.

pub async fn open(&self, path: &Path) -> Result<Box<dyn Reader>, Error>

Open a file for path.

Parameters

• path: Absolute path to the file.

pub async fn open_with_size( &self, path: &Path, known_size: usize, ) -> Result<Box<dyn Reader>, Error>

Open a reader for a file with known size.

This size may either have been retrieved from a list operation or cached metadata. By passing in the known size, we can skip a HEAD / metadata call.

pub async fn create_local_writer(path: &Path) -> Result<ObjectWriter, Error>

Create an ObjectWriter from local std::path::Path

pub async fn open_local(path: &Path) -> Result<Box<dyn Reader>, Error>

Open an Reader from local std::path::Path

pub async fn create(&self, path: &Path) -> Result<Box<dyn Writer>, Error>

Create a new file.

pub async fn put( &self, path: &Path, content: &[u8], ) -> Result<WriteResult, Error>

A helper function to create a file and write content to it.

pub async fn delete(&self, path: &Path) -> Result<(), Error>

pub async fn copy(&self, from: &Path, to: &Path) -> Result<(), Error>

pub async fn read_dir( &self, dir_path: impl Into<Path>, ) -> Result<Vec<String>, Error>

Read a directory (start from base directory) and returns all sub-paths in the directory.

pub fn list( &self, path: Option<Path>, ) -> Pin<Box<dyn Stream<Item = Result<ObjectMeta, Error>> + Send>>

pub fn read_dir_all<'a, 'b>( &'a self, dir_path: impl Into<&'b Path> + Send, unmodified_since: Option<DateTime<Utc>>, ) -> Pin<Box<dyn Stream<Item = Result<ObjectMeta, Error>> + Send + 'a>>

Read all files (start from base directory) recursively

unmodified_since can be specified to only return files that have not been modified since the given time.

pub async fn remove_dir_all( &self, dir_path: impl Into<Path>, ) -> Result<(), Error>

Remove a directory recursively.

pub fn remove_stream<'a>( &'a self, locations: Pin<Box<dyn Stream<Item = Result<Path, Error>> + Send + 'a>>, ) -> Pin<Box<dyn Stream<Item = Result<Path, Error>> + Send + 'a>>

pub async fn exists(&self, path: &Path) -> Result<bool, Error>

Check a file exists.

pub async fn size(&self, path: &Path) -> Result<u64, Error>

Get file size.

pub async fn read_one_all(&self, path: &Path) -> Result<Bytes, Error>

Convenience function to open a reader and read all the bytes

pub async fn read_one_range( &self, path: &Path, range: Range<usize>, ) -> Result<Bytes, Error>

Convenience function open a reader and make a single request

If you will be making multiple requests to the path it is more efficient to call Self::open and then call Reader::get_range multiple times.

impl ObjectStore

pub fn new( store: Arc<dyn ObjectStore>, location: Url, block_size: Option<usize>, wrapper: Option<Arc<dyn WrappingObjectStore>>, use_constant_size_upload_parts: bool, list_is_lexically_ordered: bool, io_parallelism: usize, download_retry_count: usize, storage_options: Option<&HashMap<String, String>>, ) -> ObjectStore

Trait Implementations

impl Clone for ObjectStore

fn clone(&self) -> ObjectStore

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ObjectStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl DeepSizeOf for ObjectStore

fn deep_size_of_children(&self, context: &mut Context) -> usize

Returns an estimation of the heap-managed storage of this object. This does not include the size of the object itself.

fn deep_size_of(&self) -> usize

Returns an estimation of a total size of memory owned by the object, including heap-managed storage.

impl Display for ObjectStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.