Trait StorageHandler 

pub trait StorageHandler: AsAny {
    // Required methods
    fn list_from(
        &self,
        path: &Url,
    ) -> DeltaResult<Box<dyn Iterator<Item = DeltaResult<FileMeta>>>>;
    fn read_files(
        &self,
        files: Vec<FileSlice>,
    ) -> DeltaResult<Box<dyn Iterator<Item = DeltaResult<Bytes>>>>;
    fn copy_atomic(&self, src: &Url, dest: &Url) -> DeltaResult<()>;
    fn put(&self, path: &Url, data: Bytes, overwrite: bool) -> DeltaResult<()>;
    fn head(&self, path: &Url) -> DeltaResult<FileMeta>;
    fn delete(&self, path: &Url) -> DeltaResult<()>;

    // Provided methods
    fn list_from_with_cancellation(
        &self,
        path: &Url,
        cancellation_token: Option<CancellationTokenRef>,
    ) -> DeltaResult<Box<dyn Iterator<Item = DeltaResult<FileMeta>>>> { ... }
    fn read_files_with_cancellation(
        &self,
        files: Vec<FileSlice>,
        cancellation_token: Option<CancellationTokenRef>,
    ) -> DeltaResult<Box<dyn Iterator<Item = DeltaResult<Bytes>>>> { ... }
}

Provides file system related functionalities to Delta Kernel.

Delta Kernel uses this handler whenever it needs to access the underlying file system where the Delta table is present. Connector implementation of this trait can hide filesystem specific details from Delta Kernel.

Required Methods

fn list_from( &self, path: &Url, ) -> DeltaResult<Box<dyn Iterator<Item = DeltaResult<FileMeta>>>>

Recursively list files whose full path is lexicographically greater than (UTF-8 sorting) the given path, restricted to descendants of path’s parent directory. The result must be sorted by the full path (UTF-8 byte order).

The listing is recursive: files in nested subdirectories are included, not just files directly under the parent. For example, listing from dir/0001.json may return dir/0002.json, dir/sub/0003.json, and dir/sub/nested/0004.json, all interleaved in lexicographic order.

The parent directory is derived from path:

• If path is directory-like (ends with /), the parent is path itself and the result contains all files at or below that directory.
• Otherwise, the parent is the directory containing path, and only files (at any depth under that parent) whose full path sorts strictly greater than path are returned.

fn read_files( &self, files: Vec<FileSlice>, ) -> DeltaResult<Box<dyn Iterator<Item = DeltaResult<Bytes>>>>

Read data specified by the start and end offset from the file.

fn copy_atomic(&self, src: &Url, dest: &Url) -> DeltaResult<()>

Copy a file atomically from source to destination. If the destination file already exists, it must return Err(Error::FileAlreadyExists).

fn put(&self, path: &Url, data: Bytes, overwrite: bool) -> DeltaResult<()>

Write data to the specified path.

If overwrite is false and the file already exists, this must return Err(Error::FileAlreadyExists).

fn head(&self, path: &Url) -> DeltaResult<FileMeta>

Perform a HEAD request for the given file at a Url, returning the file metadata.

If the file does not exist, this must return an Err with Error::FileNotFound.

fn delete(&self, path: &Url) -> DeltaResult<()>

Delete the file at the given path.

This operation is idempotent: deleting a path that does not exist should return Ok(()). For any other error, this must propagate the corresponding error.

Provided Methods

fn list_from_with_cancellation( &self, path: &Url, cancellation_token: Option<CancellationTokenRef>, ) -> DeltaResult<Box<dyn Iterator<Item = DeltaResult<FileMeta>>>>

Cancellation-aware variant of list_from.

When cancellation_token is Some, an engine may race its listing against the token and terminate the returned iterator with Error::Cancelled once cancellation is observed, rather than paging through the whole listing.

The default implementation returns Error::Cancelled if the token is already cancelled and otherwise delegates to list_from, ignoring the token for the rest of the listing. So an engine that does not override this stays source-compatible while still honoring an up-front cancellation; kernel additionally polls the token as it consumes the listing. An engine that overrides this takes over the up-front check and should also fast-path an already-cancelled token before starting I/O.

fn read_files_with_cancellation( &self, files: Vec<FileSlice>, cancellation_token: Option<CancellationTokenRef>, ) -> DeltaResult<Box<dyn Iterator<Item = DeltaResult<Bytes>>>>

Cancellation-aware variant of read_files.

When cancellation_token is Some, an engine may race its I/O against the token and terminate the returned iterator with Error::Cancelled once cancellation is observed, rather than reading every file slice to completion.

The default implementation returns Error::Cancelled if the token is already cancelled and otherwise delegates to read_files, ignoring the token for the rest of the read. So an engine that does not override this stays source-compatible while still honoring an up-front cancellation. An engine that overrides this takes over the up-front check and should also fast-path an already-cancelled token before starting I/O.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl StorageHandler for MeteredStorageHandler

impl StorageHandler for PlanBasedStorageHandler

Available on crate feature declarative-plans and crate feature default-engine-base and (crate features arrow-conversion or declarative-plans or default-engine-base) only.