Struct datafusion::datasource::listing::ListingOptions
source · [−]pub struct ListingOptions {
pub file_extension: String,
pub format: Arc<dyn FileFormat>,
pub table_partition_cols: Vec<String>,
pub collect_stat: bool,
pub target_partitions: usize,
}
Expand description
Options for creating a ListingTable
Fields
file_extension: String
A suffix on which files should be filtered (leave empty to keep all files on the path)
format: Arc<dyn FileFormat>
The file format
table_partition_cols: Vec<String>
The expected partition column names in the folder structure.
For example Vec["a", "b"]
means that the two first levels of
partitioning expected should be named “a” and “b”:
- If there is a third level of partitioning it will be ignored.
- Files that don’t follow this partitioning will be ignored.
Note that only
DEFAULT_PARTITION_COLUMN_DATATYPE
is currently supported for the column type.
collect_stat: bool
Set true to try to guess statistics from the files. This can add a lot of overhead as it will usually require files to be opened and at least partially parsed.
target_partitions: usize
Group files to avoid that the number of partitions exceeds this limit
Implementations
sourceimpl ListingOptions
impl ListingOptions
sourcepub fn new(format: Arc<dyn FileFormat>) -> Self
pub fn new(format: Arc<dyn FileFormat>) -> Self
Creates an options instance with the given format Default values:
- no file extension filter
- no input partition to discover
- one target partition
- no stat collection
sourcepub async fn infer_schema<'a>(
&'a self,
object_store: Arc<dyn ObjectStore>,
path: &'a str
) -> Result<SchemaRef>
pub async fn infer_schema<'a>(
&'a self,
object_store: Arc<dyn ObjectStore>,
path: &'a str
) -> Result<SchemaRef>
Infer the schema of the files at the given path on the provided object store. The inferred schema does not include the partitioning columns.
This method will not be called by the table itself but before creating it. This way when creating the logical plan we can decide to resolve the schema locally or ask a remote service to do it (e.g a scheduler).
Trait Implementations
sourceimpl Clone for ListingOptions
impl Clone for ListingOptions
sourcefn clone(&self) -> ListingOptions
fn clone(&self) -> ListingOptions
Returns a copy of the value. Read more
1.0.0 · sourcefn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from source
. Read more
Auto Trait Implementations
impl !RefUnwindSafe for ListingOptions
impl Send for ListingOptions
impl Sync for ListingOptions
impl Unpin for ListingOptions
impl !UnwindSafe for ListingOptions
Blanket Implementations
sourceimpl<T> BorrowMut<T> for T where
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
sourceimpl<T> ToOwned for T where
T: Clone,
impl<T> ToOwned for T where
T: Clone,
type Owned = T
type Owned = T
The resulting type after obtaining ownership.
sourcefn clone_into(&self, target: &mut T)
fn clone_into(&self, target: &mut T)
toowned_clone_into
)Uses borrowed data to replace owned data, usually by cloning. Read more