Struct VortexSource 

pub struct VortexSource { /* private fields */ }

File scan implementation for reading one or more .vortex files.

VortexSource is the lower-level read component underneath VortexFormat. It is the type DataFusion stores in a FileScanConfig, and it is ultimately executed through DataSourceExec.

            ▲
            │
            │  Produce a stream of
            │  RecordBatches
            │
┌───────────────────────┐
│     DataSourceExec    │
└───────────────────────┘
            ▲
            │ uses
            │
┌───────────────────────┐
│      VortexSource     │
└───────────────────────┘
            ▲
            │ opens `.vortex` files via
            │
       ObjectStore / VortexReadAt

Most applications reach VortexSource indirectly through VortexFormatFactory. Use VortexSource directly when you are constructing a FileScanConfig yourself or when you need to inject lower-level behavior such as a custom VortexReaderFactory, an external VortexAccessPlan, or a specific FileMetadataCache.

Example

use std::sync::Arc;

use arrow_schema::Schema;
use datafusion_datasource::file_scan_config::FileScanConfigBuilder;
use datafusion_datasource::source::DataSourceExec;
use datafusion_datasource::PartitionedFile;
use datafusion_datasource::TableSchema;
use datafusion_execution::object_store::ObjectStoreUrl;
use vortex::VortexSessionDefault;
use vortex::session::VortexSession;
use vortex_datafusion::VortexSource;

let file_schema = Arc::new(Schema::empty());
let source = Arc::new(
    VortexSource::new(
        TableSchema::from_file_schema(file_schema),
        VortexSession::default(),
    )
    .with_projection_pushdown(true)
    .with_scan_concurrency(4),
);

let config = FileScanConfigBuilder::new(ObjectStoreUrl::local_filesystem(), source)
    .with_file(PartitionedFile::new("metrics.vortex", 1024))
    .build();

let exec = DataSourceExec::from_data_source(config);

What VortexSource Handles

VortexSource is responsible for:

• translating DataFusion filters into Vortex predicates when possible,
• retaining the full predicate for file pruning based on statistics and partition values,
• configuring per-file readers and sharing parsed layout readers across partitions within the same scan,
• carrying the table schema used for schema evolution and missing-column adaptation,
• attaching a Vortex metrics registry to the read path.

Projection And Predicate Behavior

VortexSource keeps two related predicate forms:

• full_predicate, which is used by DataFusion’s FilePruner to skip whole files before they are opened,
• vortex_predicate, which contains only the expressions Vortex can evaluate during the scan.

Projection handling depends on VortexTableOptions::projection_pushdown:

• when disabled, VortexSource still prunes unreferenced top-level columns, but DataFusion applies the full projection after the scan,
• when enabled, the scan can evaluate a Vortex-native projection and leave only unsupported expressions for DataFusion.

Observability

VortexSource owns a Vortex metrics registry for the lifetime of a physical scan. The registry is passed to the reader and scan builder so I/O and scan metrics accumulate as the query executes.

Use VortexMetricsFinder to merge those metrics back into DataFusion MetricsSet values after the plan has run.

Execution Flow

At execution time:

1. DataFusion calls DataSourceExec, which delegates file opening to VortexSource.
2. VortexSource creates a VortexOpener configured with the current projection, predicate, options, and metrics.
3. The opener adapts filters and schema for the specific file, applies any VortexAccessPlan, and builds a Vortex scan.
4. Scan results are converted into Arrow RecordBatch values for DataFusion.

Implementations

impl VortexSource

pub fn new(table_schema: TableSchema, session: VortexSession) -> Self

Creates a new VortexSource for a table schema and VortexSession.

The new source starts with:

• all top-level columns projected,
• no pushed filters,
• a default Vortex metrics registry,
• default VortexTableOptions.

pub fn with_projection_pushdown(self, enabled: bool) -> Self

Enables or disables Vortex-native projection evaluation.

This toggles whether VortexSource tries to split DataFusion projection expressions into a Vortex scan projection plus a leftover DataFusion projection.

pub fn with_expression_convertor( self, expr_convertor: Arc<dyn ExpressionConvertor>, ) -> Self

Sets the ExpressionConvertor used to translate DataFusion expressions into Vortex expressions.

Override this when the default converter is insufficient for an engine integration or for a custom schema-adaptation strategy.

pub fn with_vortex_reader_factory( self, vortex_reader_factory: Arc<dyn VortexReaderFactory>, ) -> Self

Sets a custom factory for the underlying VortexReadAt.

Use this when reads need to go through an application-specific layer rather than the default DataFusion ObjectStore.

pub fn metrics_registry(&self) -> &Arc<dyn MetricsRegistry>

Returns the MetricsRegistry attached to this scan.

The registry is populated as files are opened and scanned. In most callers, crate::metrics::VortexMetricsFinder is the more convenient public API for turning the registry contents into DataFusion metrics.

pub fn with_file_metadata_cache( self, file_metadata_cache: Arc<dyn FileMetadataCache>, ) -> Self

Overrides the metadata cache used to reuse Vortex footers across scans.

pub fn with_scan_concurrency(self, scan_concurrency: usize) -> Self

Sets the per-file Vortex scan concurrency.

This is separate from DataFusion’s partition-level parallelism.

pub fn options(&self) -> &VortexTableOptions

Returns the effective table options for this source.

pub fn with_options(self, opts: VortexTableOptions) -> Self

Replaces the table options for this source.

Trait Implementations

impl Clone for VortexSource

fn clone(&self) -> VortexSource

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl FileSource for VortexSource

fn create_file_opener( &self, object_store: Arc<dyn ObjectStore>, base_config: &FileScanConfig, partition: usize, ) -> DFResult<Arc<dyn FileOpener>>

Creates a dyn FileOpener based on given parameters

fn as_any(&self) -> &dyn Any

Any

fn with_batch_size(&self, batch_size: usize) -> Arc<dyn FileSource>

Initialize new type with batch size configuration

fn filter(&self) -> Option<Arc<dyn PhysicalExpr>>

Returns the filter expression that will be applied during the file scan.

fn metrics(&self) -> &ExecutionPlanMetricsSet

Return execution plan metrics

fn file_type(&self) -> &str

String representation of file source such as “csv”, “json”, “parquet”

fn fmt_extra(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> Result

Format FileType specific information

fn supports_repartitioning(&self) -> bool

Returns whether this file source supports repartitioning files by byte ranges.

fn try_pushdown_filters( &self, filters: Vec<Arc<dyn PhysicalExpr>>, _config: &ConfigOptions, ) -> DFResult<FilterPushdownPropagation<Arc<dyn FileSource>>>

Try to push down filters into this FileSource.

fn try_pushdown_projection( &self, projection: &ProjectionExprs, ) -> DFResult<Option<Arc<dyn FileSource>>>

Try to push down a projection into this FileSource.

fn projection(&self) -> Option<&ProjectionExprs>

Return the projection that will be applied to the output stream on top of Self::table_schema.

fn table_schema(&self) -> &TableSchema

Returns the table schema for the overall table (including partition columns, if any)

fn repartitioned( &self, target_partitions: usize, repartition_file_min_size: usize, output_ordering: Option<LexOrdering>, config: &FileScanConfig, ) -> Result<Option<FileScanConfig>, DataFusionError>

If supported by the FileSource, redistribute files across partitions according to their size. Allows custom file formats to implement their own repartitioning logic.

fn try_pushdown_sort( &self, order: &[PhysicalSortExpr], eq_properties: &EquivalenceProperties, ) -> Result<SortOrderPushdownResult<Arc<dyn FileSource>>, DataFusionError>

Try to create a new FileSource that can produce data in the specified sort order.

fn try_reverse_output( &self, _order: &[PhysicalSortExpr], _eq_properties: &EquivalenceProperties, ) -> Result<SortOrderPushdownResult<Arc<dyn FileSource>>, DataFusionError>

👎Deprecated since 53.0.0:

Renamed to try_pushdown_sort. This method was never limited to reversing output. It will be removed in 59.0.0 or later.

Deprecated: Renamed to Self::try_pushdown_sort.

fn with_schema_adapter_factory( &self, _factory: Arc<dyn SchemaAdapterFactory>, ) -> Result<Arc<dyn FileSource>, DataFusionError>

👎Deprecated since 53.0.0:

SchemaAdapterFactory has been removed. Use PhysicalExprAdapterFactory instead. See upgrading.md for more details.

Deprecated: Set optional schema adapter factory.

fn schema_adapter_factory(&self) -> Option<Arc<dyn SchemaAdapterFactory>>

👎Deprecated since 53.0.0:

SchemaAdapterFactory has been removed. Use PhysicalExprAdapterFactory instead. See upgrading.md for more details.

Deprecated: Returns the current schema adapter factory if set.