Struct object_store::local::LocalFileSystem

source ·
pub struct LocalFileSystem { /* private fields */ }
Expand description

Local filesystem storage providing an ObjectStore interface to files on local disk. Can optionally be created with a directory prefix

§Path Semantics

This implementation follows the file URI scheme outlined in RFC 3986. In particular paths are delimited by /

§Path Semantics

LocalFileSystem will expose the path semantics of the underlying filesystem, which may have additional restrictions beyond those enforced by Path.

For example:

  • Windows forbids certain filenames, e.g. COM0,
  • Windows forbids folders with trailing .
  • Windows forbids certain ASCII characters, e.g. < or |
  • OS X forbids filenames containing :
  • Leading - are discouraged on Unix systems where they may be interpreted as CLI flags
  • Filesystems may have restrictions on the maximum path or path segment length
  • Filesystem support for non-ASCII characters is inconsistent

Additionally some filesystems, such as NTFS, are case-insensitive, whilst others like FAT don’t preserve case at all. Further some filesystems support non-unicode character sequences, such as unpaired UTF-16 surrogates, and LocalFileSystem will error on encountering such sequences.

Finally, filenames matching the regex /.*#\d+/, e.g. foo.parquet#123, are not supported by LocalFileSystem as they are used to provide atomic writes. Such files will be ignored for listing operations, and attempting to address such a file will error.

§Tokio Compatibility

Tokio discourages performing blocking IO on a tokio worker thread, however, no major operating systems have stable async file APIs. Therefore if called from a tokio context, this will use tokio::runtime::Handle::spawn_blocking to dispatch IO to a blocking thread pool, much like tokio::fs does under-the-hood.

If not called from a tokio context, this will perform IO on the current thread with no additional complexity or overheads

LocalFileSystem will follow symlinks as normal, however, it is worth noting:

  • Broken symlinks will be silently ignored by listing operations
  • No effort is made to prevent breaking symlinks when deleting files
  • Symlinks that resolve to paths outside the root will be followed
  • Mutating a file through one or more symlinks will mutate the underlying file
  • Deleting a path that resolves to a symlink will only delete the symlink

§Cross-Filesystem Copy

LocalFileSystem::copy is implemented using std::fs::hard_link, and therefore does not support copying across filesystem boundaries.

Implementations§

source§

impl LocalFileSystem

source

pub fn new() -> Self

Create new filesystem storage with no prefix

source

pub fn new_with_prefix(prefix: impl AsRef<Path>) -> Result<Self>

Create new filesystem storage with prefix applied to all paths

Returns an error if the path does not exist

source

pub fn path_to_filesystem(&self, location: &Path) -> Result<PathBuf>

Return an absolute filesystem path of the given file location

Trait Implementations§

source§

impl Debug for LocalFileSystem

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
source§

impl Default for LocalFileSystem

source§

fn default() -> Self

Returns the “default value” for a type. Read more
source§

impl Display for LocalFileSystem

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
source§

impl ObjectStore for LocalFileSystem

source§

fn put_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, payload: PutPayload, opts: PutOptions ) -> Pin<Box<dyn Future<Output = Result<PutResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Save the provided payload to location with the given options
source§

fn put_multipart_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, opts: PutMultipartOpts ) -> Pin<Box<dyn Future<Output = Result<Box<dyn MultipartUpload>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a multipart upload with options Read more
source§

fn get_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, options: GetOptions ) -> Pin<Box<dyn Future<Output = Result<GetResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a get request with options
source§

fn get_range<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, range: Range<usize> ) -> Pin<Box<dyn Future<Output = Result<Bytes>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Return the bytes that are stored at the specified location in the given byte range. Read more
source§

fn get_ranges<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, location: &'life1 Path, ranges: &'life2 [Range<usize>] ) -> Pin<Box<dyn Future<Output = Result<Vec<Bytes>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Return the bytes that are stored at the specified location in the given byte ranges
source§

fn delete<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Delete the object at the specified location.
source§

fn list(&self, prefix: Option<&Path>) -> BoxStream<'_, Result<ObjectMeta>>

List all the objects with the given prefix. Read more
source§

fn list_with_delimiter<'life0, 'life1, 'async_trait>( &'life0 self, prefix: Option<&'life1 Path> ) -> Pin<Box<dyn Future<Output = Result<ListResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

List objects with the given prefix and an implementation specific delimiter. Returns common prefixes (directories) in addition to object metadata. Read more
source§

fn copy<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Copy an object from one path to another in the same object store. Read more
source§

fn rename<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Move an object from one path to another in the same object store. Read more
source§

fn copy_if_not_exists<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Copy an object from one path to another, only if destination is empty. Read more
source§

fn put<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, payload: PutPayload ) -> Pin<Box<dyn Future<Output = Result<PutResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Save the provided bytes to the specified location Read more
source§

fn put_multipart<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path ) -> Pin<Box<dyn Future<Output = Result<Box<dyn MultipartUpload>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a multipart upload Read more
source§

fn get<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path ) -> Pin<Box<dyn Future<Output = Result<GetResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Return the bytes that are stored at the specified location.
source§

fn head<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path ) -> Pin<Box<dyn Future<Output = Result<ObjectMeta>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Return the metadata for the specified location
source§

fn delete_stream<'a>( &'a self, locations: BoxStream<'a, Result<Path>> ) -> BoxStream<'a, Result<Path>>

Delete all the objects at the specified locations Read more
source§

fn list_with_offset( &self, prefix: Option<&Path>, offset: &Path ) -> BoxStream<'_, Result<ObjectMeta>>

List all the objects with the given prefix and a location greater than offset Read more
source§

fn rename_if_not_exists<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Move an object from one path to another in the same object store. Read more

Auto Trait Implementations§

Blanket Implementations§

source§

impl<T> Any for T
where T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for T
where T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T> Instrument for T

source§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
source§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
source§

impl<T, U> Into<U> for T
where U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T> IntoEither for T

source§

fn into_either(self, into_left: bool) -> Either<Self, Self>

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
source§

impl<T> Same for T

§

type Output = T

Should always be Self
source§

impl<T> ToString for T
where T: Display + ?Sized,

source§

default fn to_string(&self) -> String

Converts the given value to a String. Read more
source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

source§

fn vzip(self) -> V

source§

impl<T> WithSubscriber for T

source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
source§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more