Struct MemorySourceConfig

pub struct MemorySourceConfig { /* private fields */ }

Data source configuration for reading in-memory batches of data

Implementations

impl MemorySourceConfig

pub fn try_new( partitions: &[Vec<RecordBatch>], schema: SchemaRef, projection: Option<Vec<usize>>, ) -> Result<Self>

Create a new MemorySourceConfig for reading in-memory record batches The provided schema should not have the projection applied.

pub fn try_new_exec( partitions: &[Vec<RecordBatch>], schema: SchemaRef, projection: Option<Vec<usize>>, ) -> Result<Arc<DataSourceExec>>

Create a new DataSourceExec plan for reading in-memory record batches The provided schema should not have the projection applied.

pub fn try_new_as_values( schema: SchemaRef, data: Vec<Vec<Arc<dyn PhysicalExpr>>>, ) -> Result<Arc<DataSourceExec>>

Create a new execution plan from a list of constant values (ValuesExec)

pub fn try_new_from_batches( schema: SchemaRef, batches: Vec<RecordBatch>, ) -> Result<Arc<DataSourceExec>>

Create a new plan using the provided schema and batches.

Errors if any of the batches don’t match the provided schema, or if no batches are provided.

pub fn with_limit(self, limit: Option<usize>) -> Self

Set the limit of the files

pub fn with_show_sizes(self, show_sizes: bool) -> Self

Set show_sizes to determine whether to display partition sizes

pub fn partitions(&self) -> &[Vec<RecordBatch>]

Ref to partitions

pub fn projection(&self) -> &Option<Vec<usize>>

Ref to projection

pub fn show_sizes(&self) -> bool

Show sizes

pub fn sort_information(&self) -> &[LexOrdering]

Ref to sort information

pub fn try_with_sort_information( self, sort_information: Vec<LexOrdering>, ) -> Result<Self>

A memory table can be ordered by multiple expressions simultaneously. EquivalenceProperties keeps track of expressions that describe the global ordering of the schema. These columns are not necessarily same; e.g.

┌-------┐
| a | b |
|---|---|
| 1 | 9 |
| 2 | 8 |
| 3 | 7 |
| 5 | 5 |
└---┴---┘

where both a ASC and b DESC can describe the table ordering. With EquivalenceProperties, we can keep track of these equivalences and treat a ASC and b DESC as the same ordering requirement.

Note that if there is an internal projection, that projection will be also applied to the given sort_information.

pub fn original_schema(&self) -> SchemaRef

Arc clone of ref to original schema

Trait Implementations

impl Clone for MemorySourceConfig

fn clone(&self) -> MemorySourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl DataSource for MemorySourceConfig

fn repartitioned( &self, target_partitions: usize, _repartition_file_min_size: usize, output_ordering: Option<LexOrdering>, ) -> Result<Option<Arc<dyn DataSource>>>

If possible, redistribute batches across partitions according to their size.

Returns Ok(None) if unable to repartition. Preserve output ordering if exists. Refer to DataSource::repartitioned for further details.

fn open( &self, partition: usize, _context: Arc<TaskContext>, ) -> Result<SendableRecordBatchStream>

fn as_any(&self) -> &dyn Any

fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> Result

Format this source for display in explain plans

fn output_partitioning(&self) -> Partitioning

fn eq_properties(&self) -> EquivalenceProperties

fn statistics(&self) -> Result<Statistics>

fn with_fetch(&self, limit: Option<usize>) -> Option<Arc<dyn DataSource>>

Return a copy of this DataSource with a new fetch limit

fn fetch(&self) -> Option<usize>

fn try_swapping_with_projection( &self, projection: &ProjectionExec, ) -> Result<Option<Arc<dyn ExecutionPlan>>>

fn metrics(&self) -> ExecutionPlanMetricsSet

fn try_pushdown_filters( &self, filters: Vec<Arc<dyn PhysicalExpr>>, _config: &ConfigOptions, ) -> Result<FilterPushdownPropagation<Arc<dyn DataSource>>>

Try to push down filters into this DataSource. See ExecutionPlan::handle_child_pushdown_result for more details.

impl Debug for MemorySourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.