Struct LocalFileSystem 

pub struct LocalFileSystem { /* private fields */ }

Available on crate feature fs and non-WebAssembly only.

Local filesystem storage providing an ObjectStore interface to files on local disk. Can optionally be created with a directory prefix

Path Semantics

This implementation follows the file URI scheme outlined in RFC 3986. In particular paths are delimited by /

Path Semantics

LocalFileSystem will expose the path semantics of the underlying filesystem, which may have additional restrictions beyond those enforced by Path.

For example:

• Windows forbids certain filenames, e.g. COM0,
• Windows forbids folders with trailing .
• Windows forbids certain ASCII characters, e.g. < or |
• OS X forbids filenames containing :
• Leading - are discouraged on Unix systems where they may be interpreted as CLI flags
• Filesystems may have restrictions on the maximum path or path segment length
• Filesystem support for non-ASCII characters is inconsistent

Additionally some filesystems, such as NTFS, are case-insensitive, whilst others like FAT don’t preserve case at all. Further some filesystems support non-unicode character sequences, such as unpaired UTF-16 surrogates, and LocalFileSystem will error on encountering such sequences.

Finally, filenames matching the regex /.*#\d+/, e.g. foo.parquet#123, are not supported by LocalFileSystem as they are used to provide atomic writes. Such files will be ignored for listing operations, and attempting to address such a file will error.

Tokio Compatibility

Tokio discourages performing blocking IO on a tokio worker thread, however, no major operating systems have stable async file APIs. Therefore if called from a tokio context, this will use tokio::runtime::Handle::spawn_blocking to dispatch IO to a blocking thread pool, much like tokio::fs does under-the-hood.

If not called from a tokio context, this will perform IO on the current thread with no additional complexity or overheads

Symlinks

LocalFileSystem will follow symlinks as normal, however, it is worth noting:

• Broken symlinks will be silently ignored by listing operations
• No effort is made to prevent breaking symlinks when deleting files
• Symlinks that resolve to paths outside the root will be followed
• Mutating a file through one or more symlinks will mutate the underlying file
• Deleting a path that resolves to a symlink will only delete the symlink

Cross-Filesystem Copy

LocalFileSystem::copy_opts is implemented using std::fs::hard_link, and therefore does not support copying across filesystem boundaries.

Implementations

impl LocalFileSystem

pub fn new() -> Self

Create new filesystem storage with no prefix

pub fn new_with_prefix(prefix: impl AsRef<Path>) -> Result<Self>

Create new filesystem storage with prefix applied to all paths

Returns an error if the path does not exist

pub fn path_to_filesystem(&self, location: &Path) -> Result<PathBuf>

Return an absolute filesystem path of the given file location

pub fn with_automatic_cleanup(self, automatic_cleanup: bool) -> Self

Enable automatic cleanup of empty directories when deleting files

pub fn with_fsync(self, fsync: bool) -> Self

Enable fsync after writes for durability

When enabled, LocalFileSystem calls File::sync_all on written files and fsyncs the affected parent directories before a write operation (put_opts, copy_opts, rename_opts, and multipart upload completion) returns success. This guarantees that both the file contents and the directory entries pointing to them are durable on stable storage, matching the implicit durability contract of remote object stores such as S3 or GCS.

This trades write throughput for durability and is disabled by default.

Note that directory fsync is only performed on Unix; on other platforms (e.g. Windows) it is a no-op, as directories cannot be portably opened and synced.

Trait Implementations

impl Clone for LocalFileSystem

fn clone(&self) -> LocalFileSystem

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LocalFileSystem

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for LocalFileSystem

fn default() -> Self

Returns the “default value” for a type.

impl Display for LocalFileSystem

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl ObjectStore for LocalFileSystem

fn put_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, payload: PutPayload, opts: PutOptions, ) -> Pin<Box<dyn Future<Output = Result<PutResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Save the provided payload to location with the given options

fn put_multipart_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, opts: PutMultipartOptions, ) -> Pin<Box<dyn Future<Output = Result<Box<dyn MultipartUpload>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a multipart upload with options

fn get_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, options: GetOptions, ) -> Pin<Box<dyn Future<Output = Result<GetResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a get request with options

fn get_ranges<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, location: &'life1 Path, ranges: &'life2 [Range<u64>], ) -> Pin<Box<dyn Future<Output = Result<Vec<Bytes>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Return the bytes that are stored at the specified location in the given byte ranges

fn delete_stream( &self, locations: BoxStream<'static, Result<Path>>, ) -> BoxStream<'static, Result<Path>>

Delete all the objects at the specified locations

fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>>

List all the objects with the given prefix.

fn list_with_offset( &self, prefix: Option<&Path>, offset: &Path, ) -> BoxStream<'static, Result<ObjectMeta>>

List all the objects with the given prefix and a location greater than offset

fn list_with_delimiter<'life0, 'life1, 'async_trait>( &'life0 self, prefix: Option<&'life1 Path>, ) -> Pin<Box<dyn Future<Output = Result<ListResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

List objects with the given prefix and an implementation specific delimiter. Returns common prefixes (directories) in addition to object metadata.

fn copy_opts<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path, options: CopyOptions, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Copy an object from one path to another in the same object store.

fn rename_opts<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path, options: RenameOptions, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Move an object from one path to another in the same object store.