Struct MemorySourceConfig 

pub struct MemorySourceConfig { /* private fields */ }

Data source configuration for reading in-memory batches of data

Implementations

impl MemorySourceConfig

pub fn try_new( partitions: &[Vec<RecordBatch>], schema: Arc<Schema>, projection: Option<Vec<usize>>, ) -> Result<MemorySourceConfig, DataFusionError>

Create a new MemorySourceConfig for reading in-memory record batches The provided schema should not have the projection applied.

pub fn try_new_exec( partitions: &[Vec<RecordBatch>], schema: Arc<Schema>, projection: Option<Vec<usize>>, ) -> Result<Arc<DataSourceExec>, DataFusionError>

Create a new DataSourceExec plan for reading in-memory record batches The provided schema should not have the projection applied.

pub fn try_new_as_values( schema: Arc<Schema>, data: Vec<Vec<Arc<dyn PhysicalExpr>>>, ) -> Result<Arc<DataSourceExec>, DataFusionError>

Create a new execution plan from a list of constant values (ValuesExec)

pub fn try_new_from_batches( schema: Arc<Schema>, batches: Vec<RecordBatch>, ) -> Result<Arc<DataSourceExec>, DataFusionError>

Create a new plan using the provided schema and batches.

Errors if any of the batches don’t match the provided schema, or if no batches are provided.

pub fn with_limit(self, limit: Option<usize>) -> MemorySourceConfig

Set the limit of the files

pub fn with_show_sizes(self, show_sizes: bool) -> MemorySourceConfig

Set show_sizes to determine whether to display partition sizes

pub fn partitions(&self) -> &[Vec<RecordBatch>]

Ref to partitions

pub fn projection(&self) -> &Option<Vec<usize>>

Ref to projection

pub fn show_sizes(&self) -> bool

Show sizes

pub fn sort_information(&self) -> &[LexOrdering]

Ref to sort information

pub fn try_with_sort_information( self, sort_information: Vec<LexOrdering>, ) -> Result<MemorySourceConfig, DataFusionError>

A memory table can be ordered by multiple expressions simultaneously. EquivalenceProperties keeps track of expressions that describe the global ordering of the schema. These columns are not necessarily same; e.g.

┌-------┐
| a | b |
|---|---|
| 1 | 9 |
| 2 | 8 |
| 3 | 7 |
| 5 | 5 |
└---┴---┘

where both a ASC and b DESC can describe the table ordering. With EquivalenceProperties, we can keep track of these equivalences and treat a ASC and b DESC as the same ordering requirement.

Note that if there is an internal projection, that projection will be also applied to the given sort_information.

pub fn original_schema(&self) -> Arc<Schema>

Arc clone of ref to original schema

Trait Implementations

impl Clone for MemorySourceConfig

fn clone(&self) -> MemorySourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl DataSource for MemorySourceConfig

fn repartitioned( &self, target_partitions: usize, _repartition_file_min_size: usize, output_ordering: Option<LexOrdering>, ) -> Result<Option<Arc<dyn DataSource>>, DataFusionError>

If possible, redistribute batches across partitions according to their size.

Returns Ok(None) if unable to repartition. Preserve output ordering if exists. Refer to DataSource::repartitioned for further details.

fn open( &self, partition: usize, _context: Arc<TaskContext>, ) -> Result<Pin<Box<dyn RecordBatchStream<Item = Result<RecordBatch, DataFusionError>> + Send>>, DataFusionError>

fn as_any(&self) -> &(dyn Any + 'static)

fn fmt_as( &self, t: DisplayFormatType, f: &mut Formatter<'_>, ) -> Result<(), Error>

Format this source for display in explain plans

fn output_partitioning(&self) -> Partitioning

fn eq_properties(&self) -> EquivalenceProperties

fn scheduling_type(&self) -> SchedulingType

fn partition_statistics( &self, partition: Option<usize>, ) -> Result<Statistics, DataFusionError>

Returns statistics for a specific partition, or aggregate statistics across all partitions if partition is None.

fn with_fetch(&self, limit: Option<usize>) -> Option<Arc<dyn DataSource>>

Return a copy of this DataSource with a new fetch limit

fn fetch(&self) -> Option<usize>

fn try_swapping_with_projection( &self, projection: &ProjectionExprs, ) -> Result<Option<Arc<dyn DataSource>>, DataFusionError>

fn statistics(&self) -> Result<Statistics, DataFusionError>

👎Deprecated since 51.0.0: Use partition_statistics instead

Returns aggregate statistics across all partitions.

fn metrics(&self) -> ExecutionPlanMetricsSet

fn try_pushdown_filters( &self, filters: Vec<Arc<dyn PhysicalExpr>>, _config: &ConfigOptions, ) -> Result<FilterPushdownPropagation<Arc<dyn DataSource>>, DataFusionError>

Try to push down filters into this DataSource. See ExecutionPlan::handle_child_pushdown_result for more details.

fn try_pushdown_sort( &self, _order: &[PhysicalSortExpr], ) -> Result<SortOrderPushdownResult<Arc<dyn DataSource>>, DataFusionError>

Try to create a new DataSource that produces data in the specified sort order.

impl Debug for MemorySourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.