Struct datafusion::datasource::memory::MemTable

pub struct MemTable {
    pub sort_order: Arc<Mutex<Vec<Vec<Expr>>>>,
    /* private fields */
}

In-memory data source for presenting a Vec<RecordBatch> as a data source that can be queried by DataFusion. This allows data to be pre-loaded into memory and then repeatedly queried without incurring additional file I/O overhead.

Fields

sort_order: Arc<Mutex<Vec<Vec<Expr>>>>

Optional pre-known sort order(s). Must be SortExprs. inserting data into this table removes the order

Implementations

impl MemTable

pub fn try_new( schema: SchemaRef, partitions: Vec<Vec<RecordBatch>> ) -> Result<Self>

Create a new in-memory table from the provided schema and record batches

pub fn with_constraints(self, constraints: Constraints) -> Self

Assign constraints

pub fn with_column_defaults( self, column_defaults: HashMap<String, Expr> ) -> Self

Assign column defaults

pub fn with_sort_order(self, sort_order: Vec<Vec<Expr>>) -> Self

Specify an optional pre-known sort order(s). Must be SortExprs.

If the data is not sorted by this order, DataFusion may produce incorrect results.

DataFusion may take advantage of this ordering to omit sorts or use more efficient algorithms.

Note that multiple sort orders are supported, if some are known to be equivalent,

pub async fn load( t: Arc<dyn TableProvider>, output_partitions: Option<usize>, state: &SessionState ) -> Result<Self>

Create a mem table by reading from another data source

Trait Implementations

impl Debug for MemTable

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TableProvider for MemTable

fn insert_into<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 SessionState, input: Arc<dyn ExecutionPlan>, overwrite: bool ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Returns an ExecutionPlan that inserts the execution results of a given ExecutionPlan into this MemTable.

The ExecutionPlan must have the same schema as this MemTable.

Arguments

• state - The SessionState containing the context for executing the plan.
• input - The ExecutionPlan to execute and insert.

Returns

• A plan that returns the number of rows written.

fn as_any(&self) -> &dyn Any

Returns the table provider as Any so that it can be downcast to a specific implementation.

fn schema(&self) -> SchemaRef

Get a reference to the schema for this table

fn constraints(&self) -> Option<&Constraints>

Get a reference to the constraints of the table. Returns:

fn table_type(&self) -> TableType

Get the type of this table for metadata/catalog purposes.

fn scan<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, state: &'life1 SessionState, projection: Option<&'life2 Vec<usize>>, _filters: &'life3 [Expr], _limit: Option<usize> ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Create an ExecutionPlan for scanning the table with optionally specified projection, filter and limit, described below.

fn get_column_default(&self, column: &str) -> Option<&Expr>

Get the default value for a column, if available.

fn get_table_definition(&self) -> Option<&str>

Get the create statement used to create this table, if available.

fn get_logical_plan(&self) -> Option<&LogicalPlan>

Get the LogicalPlan of this table, if available

fn supports_filter_pushdown( &self, _filter: &Expr ) -> Result<TableProviderFilterPushDown>

👎Deprecated since 20.0.0: use supports_filters_pushdown instead

Tests whether the table provider can make use of a filter expression to optimise data retrieval.

fn supports_filters_pushdown( &self, filters: &[&Expr] ) -> Result<Vec<TableProviderFilterPushDown>>

Tests whether the table provider can make use of any or all filter expressions to optimise data retrieval. Note: the returned vector much have the same size as the filters argument.

fn statistics(&self) -> Option<Statistics>

Get statistics for this table, if available