Struct deltalake::datafusion::datasource::listing::ListingTableUrl

pub struct ListingTableUrl { /* private fields */ }

A parsed URL identifying files for a listing table, see ListingTableUrl::parse for more information on the supported expressions

Implementations

impl ListingTableUrl

pub fn parse(s: impl AsRef<str>) -> Result<ListingTableUrl, DataFusionError>

Parse a provided string as a ListingTableUrl

A URL can either refer to a single object, or a collection of objects with a common prefix, with the presence of a trailing / indicating a collection.

For example, file:///foo.txt refers to the file at /foo.txt, whereas file:///foo/ refers to all the files under the directory /foo and its subdirectories.

Similarly s3://BUCKET/blob.csv refers to blob.csv in the S3 bucket BUCKET, wherease s3://BUCKET/foo/ refers to all objects with the prefix foo/ in the S3 bucket BUCKET

URL Encoding

URL paths are expected to be URL-encoded. That is, the URL for a file named bar%2Efoo would be file:///bar%252Efoo, as per the URL specification.

It should be noted that some tools, such as the AWS CLI, take a different approach and instead interpret the URL path verbatim. For example the object bar%2Efoo would be addressed as s3://BUCKET/bar%252Efoo using ListingTableUrl but s3://BUCKET/bar%2Efoo when using the aws-cli.

Paths without a Scheme

If no scheme is provided, or the string is an absolute filesystem path as determined by std::path::Path::is_absolute, the string will be interpreted as a path on the local filesystem using the operating system’s standard path delimiter, i.e. \ on Windows, / on Unix.

If the path contains any of '?', '*', '[', it will be considered a glob expression and resolved as described in the section below.

Otherwise, the path will be resolved to an absolute path based on the current working directory, and converted to a file URI.

If the path already exists in the local filesystem this will be used to determine if this ListingTableUrl refers to a collection or a single object, otherwise the presence of a trailing path delimiter will be used to indicate a directory. For the avoidance of ambiguity it is recommended users always include trailing / when intending to refer to a directory.

Glob File Paths

If no scheme is provided, and the path contains a glob expression, it will be resolved as follows.

The string up to the first path segment containing a glob expression will be extracted, and resolved in the same manner as a normal scheme-less path above.

The remaining string will be interpreted as a glob::Pattern and used as a filter when listing files from object storage

pub fn scheme(&self) -> &str

Returns the URL scheme

pub fn prefix(&self) -> &Path

Return the URL path not excluding any glob expression

If Self::is_collection, this is the listing prefix Otherwise, this is the path to the object

pub fn contains(&self, path: &Path, ignore_subdirectory: bool) -> bool

Returns true if path matches this ListingTableUrl

pub fn is_collection(&self) -> bool

Returns true if path refers to a collection of objects

pub async fn list_all_files<'a>( &'a self, ctx: &'a SessionState, store: &'a (dyn ObjectStore + 'static), file_extension: &'a str ) -> Result<Pin<Box<dyn Stream<Item = Result<ObjectMeta, DataFusionError>> + Send + 'a>>, DataFusionError>

List all files identified by this ListingTableUrl for the provided file_extension

pub fn as_str(&self) -> &str

Returns this ListingTableUrl as a string

pub fn object_store(&self) -> ObjectStoreUrl

Return the ObjectStoreUrl for this ListingTableUrl

Trait Implementations

impl AsRef<Url> for ListingTableUrl

fn as_ref(&self) -> &Url

Converts this type into a shared reference of the (usually inferred) input type.

impl AsRef<str> for ListingTableUrl

fn as_ref(&self) -> &str

Converts this type into a shared reference of the (usually inferred) input type.

impl Clone for ListingTableUrl

fn clone(&self) -> ListingTableUrl

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ListingTableUrl

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Display for ListingTableUrl

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Hash for ListingTableUrl

fn hash<__H>(&self, state: &mut __H)

where __H: Hasher,

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for ListingTableUrl

fn eq(&self, other: &ListingTableUrl) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for ListingTableUrl

impl StructuralPartialEq for ListingTableUrl