Struct VortexSource 

pub struct VortexSource { /* private fields */ }

Execution plan for reading one or more Vortex files, intended to be consumed by DataSourceExec.

Implementations

impl VortexSource

pub fn new(table_schema: TableSchema, session: VortexSession) -> Self

Creates a new VortexSource with default configuration and a provided VortexSession. Meant to be use with a FileScanConfig to scan a file with the provided schema.

Can be configured using the provided methods.

pub fn with_projection_pushdown(self, enabled: bool) -> Self

Enable or disable expression pushdown into the underlying Vortex scan.

pub fn with_expression_convertor( self, expr_convertor: Arc<dyn ExpressionConvertor>, ) -> Self

Set a ExpressionConvertor to control how Datafusion expression should be converted and pushed down.

pub fn with_vortex_reader_factory( self, vortex_reader_factory: Arc<dyn VortexReaderFactory>, ) -> Self

Set a user-defined factory to create the underlying VortexReadAt

pub fn metrics_registry(&self) -> &Arc<dyn MetricsRegistry>

Returns the MetricsRegistry attached to this source.

pub fn with_file_metadata_cache( self, file_metadata_cache: Arc<dyn FileMetadataCache>, ) -> Self

Override the file metadata cache

pub fn with_scan_concurrency(self, scan_concurrency: usize) -> Self

Set the underlying scan concurrency. This limit is used per Vortex scan operations.

pub fn options(&self) -> &VortexTableOptions

Returns the table options for this source.

pub fn with_options(self, opts: VortexTableOptions) -> Self

Set the table options for this source.

Trait Implementations

impl Clone for VortexSource

fn clone(&self) -> VortexSource

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl FileSource for VortexSource

fn create_file_opener( &self, object_store: Arc<dyn ObjectStore>, base_config: &FileScanConfig, partition: usize, ) -> DFResult<Arc<dyn FileOpener>>

Creates a dyn FileOpener based on given parameters

fn as_any(&self) -> &dyn Any

Any

fn with_batch_size(&self, batch_size: usize) -> Arc<dyn FileSource>

Initialize new type with batch size configuration

fn filter(&self) -> Option<Arc<dyn PhysicalExpr>>

Returns the filter expression that will be applied during the file scan.

fn metrics(&self) -> &ExecutionPlanMetricsSet

Return execution plan metrics

fn file_type(&self) -> &str

String representation of file source such as “csv”, “json”, “parquet”

fn fmt_extra(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> Result

Format FileType specific information

fn supports_repartitioning(&self) -> bool

Returns whether this file source supports repartitioning files by byte ranges.

fn try_pushdown_filters( &self, filters: Vec<Arc<dyn PhysicalExpr>>, _config: &ConfigOptions, ) -> DFResult<FilterPushdownPropagation<Arc<dyn FileSource>>>

Try to push down filters into this FileSource. See ExecutionPlan::handle_child_pushdown_result for more details.

fn try_pushdown_projection( &self, projection: &ProjectionExprs, ) -> DFResult<Option<Arc<dyn FileSource>>>

Try to push down a projection into a this FileSource.

fn projection(&self) -> Option<&ProjectionExprs>

Return the projection that will be applied to the output stream on top of the table schema.

fn table_schema(&self) -> &TableSchema

Returns the table schema for this file source.

fn repartitioned( &self, target_partitions: usize, repartition_file_min_size: usize, output_ordering: Option<LexOrdering>, config: &FileScanConfig, ) -> Result<Option<FileScanConfig>, DataFusionError>

If supported by the FileSource, redistribute files across partitions according to their size. Allows custom file formats to implement their own repartitioning logic.

fn try_reverse_output( &self, _order: &[PhysicalSortExpr], _eq_properties: &EquivalenceProperties, ) -> Result<SortOrderPushdownResult<Arc<dyn FileSource>>, DataFusionError>

Try to create a new FileSource that can produce data in the specified sort order.

fn with_schema_adapter_factory( &self, _factory: Arc<dyn SchemaAdapterFactory>, ) -> Result<Arc<dyn FileSource>, DataFusionError>

👎Deprecated since 52.0.0: SchemaAdapterFactory has been removed. Use PhysicalExprAdapterFactory instead. See upgrading.md for more details.

Deprecated: Set optional schema adapter factory.

fn schema_adapter_factory(&self) -> Option<Arc<dyn SchemaAdapterFactory>>

👎Deprecated since 52.0.0: SchemaAdapterFactory has been removed. Use PhysicalExprAdapterFactory instead. See upgrading.md for more details.

Deprecated: Returns the current schema adapter factory if set.