Struct PartitionedFile

pub struct PartitionedFile {
    pub object_meta: ObjectMeta,
    pub partition_values: Vec<ScalarValue>,
    pub range: Option<FileRange>,
    pub statistics: Option<Arc<Statistics>>,
    pub extensions: Option<Arc<dyn Any + Send + Sync>>,
    pub metadata_size_hint: Option<usize>,
}

A single file or part of a file that should be read, along with its schema, statistics and partition column values that need to be appended to each row.

Fields

object_meta: ObjectMeta

Path for the file (e.g. URL, filesystem path, etc)

partition_values: Vec<ScalarValue>

Values of partition columns to be appended to each row.

These MUST have the same count, order, and type than the table_partition_cols.

You may use wrap_partition_value_in_dict to wrap them if you have used wrap_partition_type_in_dict to wrap the column type.

range: Option<FileRange>

An optional file range for a more fine-grained parallel execution

statistics: Option<Arc<Statistics>>

Optional statistics that describe the data in this file if known.

DataFusion relies on these statistics for planning (in particular to sort file groups), so if they are incorrect, incorrect answers may result.

extensions: Option<Arc<dyn Any + Send + Sync>>

An optional field for user defined per object metadata

metadata_size_hint: Option<usize>

The estimated size of the parquet metadata, in bytes

Implementations

impl PartitionedFile

pub fn new(path: impl Into<String>, size: u64) -> Self

Create a simple file without metadata or partition

pub fn new_with_range(path: String, size: u64, start: i64, end: i64) -> Self

Create a file range without metadata or partition

pub fn with_metadata_size_hint(self, metadata_size_hint: usize) -> Self

Provide a hint to the size of the file metadata. If a hint is provided the reader will try and fetch the last size_hint bytes of the parquet file optimistically. Without an appropriate hint, two read may be required to fetch the metadata.

pub fn from_path(path: String) -> Result<Self>

Return a file reference from the given path

pub fn path(&self) -> &Path

Return the path of this partitioned file

pub fn with_range(self, start: i64, end: i64) -> Self

Update the file to only scan the specified range (in bytes)

pub fn with_extensions(self, extensions: Arc<dyn Any + Send + Sync>) -> Self

Update the user defined extensions for this file.

This can be used to pass reader specific information.

pub fn with_statistics(self, statistics: Arc<Statistics>) -> Self

Trait Implementations

impl Clone for PartitionedFile

fn clone(&self) -> PartitionedFile

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PartitionedFile

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<ObjectMeta> for PartitionedFile

fn from(object_meta: ObjectMeta) -> Self

Converts to this type from the input type.

impl FromIterator<PartitionedFile> for FileGroup

fn from_iter<I: IntoIterator<Item = PartitionedFile>>(iter: I) -> Self

Creates a value from an iterator.