Struct ScanParquet 

pub struct ScanParquet {
    pub files: Vec<ScanFile>,
    pub file_constant_columns: Vec<String>,
    pub schema: SchemaRef,
}

Available on crate feature declarative-plans only.

Reads Parquet files into row batches matching schema. The engine returns exactly the columns named by schema, in schema order.

Output row order is unspecified: the engine is free to read files in any order, in parallel, and to interleave rows from different files.

Column resolution

The engine iterates schema’s fields in order; for each field it produces one column of output:

1. Metadata columns: if the field is annotated as a metadata column (e.g. via StructField::create_metadata_column with MetadataColumnSpec::RowIndex), the engine populates it from the read context rather than from the Parquet file. See Metadata columns below.
2. File-constant columns: if the field’s name appears in Self::file_constant_columns, the engine broadcasts the corresponding entry from ScanFile::file_constants for the file being read (not from Parquet bytes). See File-constant columns below.
3. Data columns: otherwise the engine attempts to locate the field in the Parquet file, in this order:
   • Field ID: if the field carries a Parquet field ID via ColumnMetadataKey::ParquetFieldId metadata, match it against the Parquet column with the same field id.
   • Field name: otherwise, or if no Parquet column has the requested field id, match by column name.
   • No match: the output column is filled with NULLs when the field is nullable, or an error is returned when it is non-nullable.

Parquet columns not referenced by any schema field are ignored.

Example

Consider a schema with the following fields (none of which are metadata columns):

• Column 0: "i_logical" (integer, non-null) with field ID 1 (via ColumnMetadataKey::ParquetFieldId)
• Column 1: "s" (string, nullable) with no field ID metadata
• Column 2: "i2" (integer, nullable) with no field ID metadata

And a Parquet file containing these columns:

• Column 0: "i2" (integer, nullable) with field ID 3
• Column 1: "i" (integer, non-null) with field ID 1
• No "s" column present

Resolving each schema field in turn:

• "i_logical" matches "i" by field ID (both have ID 1).
• "s" has no matching Parquet column, so the output column is filled with NULLs.
• "i2" matches "i2" by column name (no field ID to match on).

The returned data contains exactly 3 columns in schema order: {i_logical: parquet[1], s: NULL.., i2: parquet[0]}.

Metadata columns

A field marked as a row index metadata column (via StructField::create_metadata_column with MetadataColumnSpec::RowIndex) is populated by the engine with the 0-based row position within the file (LONG, non-nullable); a file with 5 rows yields [0, 1, 2, 3, 4]. The column name is caller-chosen (commonly "row_index").

File-constant columns

Self::file_constant_columns names output fields whose values are identical for every row in a given file (for example Delta partition columns or a table-changes version). Types and nullability come from Self::schema; ScanFile::file_constants supplies the per-file literals in the same order as file_constant_columns.

File-constant columns are distinct from metadata columns, which are engine-generated (such as row index). DynamicScan::file_constant_columns is the same concept for the DynamicScan node.

Invariants

• files[i].file_constants.len() == file_constant_columns.len() for every i.
• Every name in file_constant_columns resolves to a field in schema that is not a metadata column.
• Each scalar in file_constants is compatible with its schema field’s type.

Fields

files: Vec<ScanFile>

file_constant_columns: Vec<String>

schema: SchemaRef

Trait Implementations

impl Clone for ScanParquet

fn clone(&self) -> ScanParquet

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ScanParquet

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&ScanParquet> for ScanParquetNode

fn from(node: &ScanParquet) -> Self

Converts to this type from the input type.

impl From<ScanParquet> for Operator

fn from(payload: ScanParquet) -> Self

Converts to this type from the input type.