pub struct LocalParquetFile {Show 17 fields
pub location: ParquetSourceLocation,
pub path: PathBuf,
pub resolved_source: ResolvedParquetSource,
pub stable_id: String,
pub file_size_bytes: Option<u64>,
pub row_groups: Option<Vec<usize>>,
pub row_group_bytes: Option<Vec<u64>>,
pub projected_row_group_compressed_bytes: Option<Vec<u64>>,
pub projected_row_group_uncompressed_bytes: Option<Vec<u64>>,
pub pure_filter_row_group_compressed_bytes: Option<Vec<u64>>,
pub pure_filter_row_group_uncompressed_bytes: Option<Vec<u64>>,
pub row_group_decoded_bytes: Option<Vec<u64>>,
pub row_group_rows: Option<Vec<u64>>,
pub graph_vertex_upper_bound: Option<u64>,
pub row_group_null_counts: BTreeMap<usize, Vec<Option<u64>>>,
pub reader_decimal_types: Vec<ParquetReaderDecimalType>,
pub partition_values: Vec<LiteralValue>,
/* private fields */
}Expand description
One local or object-store Parquet file plus optional row-group metadata.
Byte vectors are aligned with the selected row groups when row_groups is
present, otherwise with the frontend’s represented row-group order. None
means the metadata is unavailable; it does not mean zero.
Fields§
§location: ParquetSourceLocationExplicit source location for the Parquet file.
path: PathBufPath-shaped reader string retained for existing diagnostics and cuDF
reader APIs. Object-store URIs are represented as path-shaped strings
but must be created through object_store_uri.
resolved_source: ResolvedParquetSourceResolved reader strategy and decode contract for this file.
stable_id: StringStable identifier for diagnostics, pruning, and executor bookkeeping.
Frontends should choose a value that remains stable across equivalent plans, commonly derived from path plus object metadata.
file_size_bytes: Option<u64>Known file size in bytes, when available from frontend metadata.
row_groups: Option<Vec<usize>>Selected Parquet row-group indices.
None means no explicit row-group selection was represented. Some
limits the scan to the listed row groups. Some(vec![]) selects zero row
groups, which marks a source proven empty.
row_group_bytes: Option<Vec<u64>>Estimated or known byte sizes for selected row groups.
projected_row_group_compressed_bytes: Option<Vec<u64>>Estimated or known compressed byte sizes for projected output file columns.
projected_row_group_uncompressed_bytes: Option<Vec<u64>>Estimated or known uncompressed byte sizes for projected output file columns.
Bounded source chunk planning uses this as its primary chunk-size input when available. Columns used only by source-adjacent filters are tracked separately so they can contribute to read/decode cost without forcing retained output sizing.
pure_filter_row_group_compressed_bytes: Option<Vec<u64>>Estimated or known compressed byte sizes for pure-filter file columns.
pure_filter_row_group_uncompressed_bytes: Option<Vec<u64>>Estimated or known uncompressed byte sizes for pure-filter file columns.
row_group_decoded_bytes: Option<Vec<u64>>Estimated decoded device bytes of the source’s read columns, in physical
Parquet row-group order. Chunk planning compares this estimate with the
source-byte target; frontends derive it with
row_group_decoded_bytes_from_footer.
row_group_rows: Option<Vec<u64>>Estimated or known row counts in physical Parquet row-group order.
The vector covers every physical row group; row_groups, when present,
contains the physical IDs selected for this source.
graph_vertex_upper_bound: Option<u64>Footer-proven upper bound on distinct graph vertices in this complete edge file, when the fixture/data producer attached one.
This is not applied to a partial row-group selection or a transformed
source. LocalParquetSource::graph_vertex_upper_bound enforces that
rowset-identity boundary before admission consumes it.
row_group_null_counts: BTreeMap<usize, Vec<Option<u64>>>Per-file-column null counts by row group, keyed by physical root column index.
Inner vectors are indexed by physical row-group number. They are not
indexed by selected row-group position. None means the footer did not
prove a null count for that column chunk.
reader_decimal_types: Vec<ParquetReaderDecimalType>Per-read-column decimal storage types proven from Parquet metadata.
partition_values: Vec<LiteralValue>Values for LocalParquetSource::partition_columns in source order.
Implementations§
Source§impl LocalParquetFile
impl LocalParquetFile
Sourcepub fn new(path: impl Into<PathBuf>) -> Self
pub fn new(path: impl Into<PathBuf>) -> Self
Creates a file descriptor with no optional metadata.
Sourcepub fn object_store_uri(uri: impl Into<String>) -> Result<Self, String>
pub fn object_store_uri(uri: impl Into<String>) -> Result<Self, String>
Creates an explicit object-store file descriptor.
Sourcepub fn from_resolved_parquet_source(
resolved_source: ResolvedParquetSource,
) -> Result<Self, String>
pub fn from_resolved_parquet_source( resolved_source: ResolvedParquetSource, ) -> Result<Self, String>
Creates a file descriptor from an already resolved reader contract.
Sourcepub fn with_resolved_parquet_source(
self,
resolved_source: ResolvedParquetSource,
) -> Result<Self, String>
pub fn with_resolved_parquet_source( self, resolved_source: ResolvedParquetSource, ) -> Result<Self, String>
Replaces the resolved reader contract and updates the reader path.
Sourcepub fn with_stable_id(self, stable_id: impl Into<String>) -> Self
pub fn with_stable_id(self, stable_id: impl Into<String>) -> Self
Sets the stable file identifier used by native diagnostics/execution.
Sourcepub fn with_file_size_bytes(self, file_size_bytes: u64) -> Self
pub fn with_file_size_bytes(self, file_size_bytes: u64) -> Self
Sets known file size in bytes.
Sourcepub fn with_row_groups(self, row_groups: Vec<usize>) -> Self
pub fn with_row_groups(self, row_groups: Vec<usize>) -> Self
Sets the selected row groups for this file.
Sourcepub fn with_row_group_bytes(self, row_group_bytes: Vec<u64>) -> Self
pub fn with_row_group_bytes(self, row_group_bytes: Vec<u64>) -> Self
Sets byte-size metadata for selected row groups.
Sourcepub fn with_projected_row_group_bytes(
self,
compressed_bytes: Vec<u64>,
uncompressed_bytes: Vec<u64>,
) -> Self
pub fn with_projected_row_group_bytes( self, compressed_bytes: Vec<u64>, uncompressed_bytes: Vec<u64>, ) -> Self
Sets projected output column byte-size metadata for row groups.
Sourcepub fn with_pure_filter_row_group_bytes(
self,
compressed_bytes: Vec<u64>,
uncompressed_bytes: Vec<u64>,
) -> Self
pub fn with_pure_filter_row_group_bytes( self, compressed_bytes: Vec<u64>, uncompressed_bytes: Vec<u64>, ) -> Self
Sets pure-filter column byte-size metadata for row groups.
Sourcepub fn with_row_group_decoded_bytes(self, decoded_bytes: Vec<u64>) -> Self
pub fn with_row_group_decoded_bytes(self, decoded_bytes: Vec<u64>) -> Self
Sets the decoded read-column byte estimate in physical Parquet row-group order.
Sourcepub fn with_row_group_rows(self, row_group_rows: Vec<u64>) -> Self
pub fn with_row_group_rows(self, row_group_rows: Vec<u64>) -> Self
Sets row-count metadata in physical Parquet row-group order.
Sourcepub fn with_graph_vertex_upper_bound(
self,
graph_vertex_upper_bound: u64,
) -> Self
pub fn with_graph_vertex_upper_bound( self, graph_vertex_upper_bound: u64, ) -> Self
Attaches a footer-proven graph vertex upper bound for this complete edge-file rowset.
Frontend code should populate this only by reading
GRAPH_VERTEX_UPPER_BOUND_PARQUET_METADATA_KEY from this file’s
Parquet footer. The bound stays on the file descriptor, so it travels with
the file it describes and cannot come from a detached query setting.
Sourcepub fn with_graph_simple_directed_edge_topology(
self,
graph_simple_directed_edge_topology: GraphSimpleDirectedEdgeTopology,
) -> Self
pub fn with_graph_simple_directed_edge_topology( self, graph_simple_directed_edge_topology: GraphSimpleDirectedEdgeTopology, ) -> Self
Attaches a footer-proven simple-directed edge topology for this complete Parquet file.
Frontends must populate this only from
GRAPH_SIMPLE_DIRECTED_EDGE_TOPOLOGY_PARQUET_METADATA_KEY. The source
contract checks complete-rowset and single-file identity before graph
lowering can use it.
Sourcepub fn with_row_group_null_counts(
self,
file_column_index: usize,
null_counts: Vec<Option<u64>>,
) -> Self
pub fn with_row_group_null_counts( self, file_column_index: usize, null_counts: Vec<Option<u64>>, ) -> Self
Sets row-group null-count metadata for one physical file column.
Sourcepub fn with_reader_decimal_types(
self,
reader_decimal_types: Vec<ParquetReaderDecimalType>,
) -> Self
pub fn with_reader_decimal_types( self, reader_decimal_types: Vec<ParquetReaderDecimalType>, ) -> Self
Sets per-read-column decimal storage types proven from Parquet metadata.
Sourcepub fn with_partition_values(self, partition_values: Vec<LiteralValue>) -> Self
pub fn with_partition_values(self, partition_values: Vec<LiteralValue>) -> Self
Sets values for represented partition columns.
Sourcepub fn selected_or_all_row_groups(&self, row_group_count: usize) -> Vec<usize>
pub fn selected_or_all_row_groups(&self, row_group_count: usize) -> Vec<usize>
Returns the selected row-group indices, or every index below
row_group_count when no selection is represented.
An explicit empty selection stays empty; see Self::row_groups.
Sourcepub fn selected_row_group_count(&self) -> usize
pub fn selected_row_group_count(&self) -> usize
Returns the represented selected row-group count, or zero if unknown.