Struct datafusion::datasource::physical_plan::FileScanConfig

pub struct FileScanConfig {
    pub object_store_url: ObjectStoreUrl,
    pub file_schema: SchemaRef,
    pub file_groups: Vec<Vec<PartitionedFile>>,
    pub statistics: Statistics,
    pub projection: Option<Vec<usize>>,
    pub limit: Option<usize>,
    pub table_partition_cols: Vec<(String, DataType)>,
    pub output_ordering: Vec<LexOrdering>,
    pub infinite_source: bool,
}

The base configurations to provide when creating a physical plan for any given file format.

Fields

object_store_url: ObjectStoreUrl

Object store URL, used to get an ObjectStore instance from RuntimeEnv::object_store

file_schema: SchemaRef

Schema before projection is applied. It contains the all columns that may appear in the files. It does not include table partition columns that may be added.

file_groups: Vec<Vec<PartitionedFile>>

List of files to be processed, grouped into partitions

Each file must have a schema of file_schema or a subset. If a particular file has a subset, the missing columns are padded with NULLs.

DataFusion may attempt to read each partition of files concurrently, however files within a partition will be read sequentially, one after the next.

statistics: Statistics

Estimated overall statistics of the files, taking filters into account.

projection: Option<Vec<usize>>

Columns on which to project the data. Indexes that are higher than the number of columns of file_schema refer to table_partition_cols.

limit: Option<usize>

The maximum number of records to read from this plan. If None, all records after filtering are returned.

table_partition_cols: Vec<(String, DataType)>

The partitioning columns

output_ordering: Vec<LexOrdering>

All equivalent lexicographical orderings that describe the schema.

infinite_source: bool

Indicates whether this plan may produce an infinite stream of records.

Implementations

impl FileScanConfig

pub fn project(&self) -> (SchemaRef, Statistics, Vec<LexOrdering>)

Project the schema and the statistics on the given column indices

pub fn repartition_file_groups( file_groups: Vec<Vec<PartitionedFile>>, target_partitions: usize, repartition_file_min_size: usize ) -> Option<Vec<Vec<PartitionedFile>>>

Repartition all input files into target_partitions partitions, if total file size exceed repartition_file_min_size target_partitions and repartition_file_min_size directly come from configuration.

This function only try to partition file byte range evenly, and let specific FileOpener to do actual partition on specific data source type. (e.g. CsvOpener will only read lines overlap with byte range but also handle boundaries to ensure all lines will be read exactly once)

Trait Implementations

impl Clone for FileScanConfig

fn clone(&self) -> FileScanConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FileScanConfig

fn fmt(&self, f: &mut Formatter<'_>) -> FmtResult

Formats the value using the given formatter.

impl DisplayAs for FileScanConfig

fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> FmtResult

Format according to DisplayFormatType, used when verbose representation looks different from the default one