Struct datafusion::datasource::memory::MemTable

pub struct MemTable { /* private fields */ }

In-memory data source for presenting a Vec<RecordBatch> as a data source that can be queried by DataFusion. This allows data to be pre-loaded into memory and then repeatedly queried without incurring additional file I/O overhead.

Implementations

impl MemTable

pub fn try_new( schema: SchemaRef, partitions: Vec<Vec<RecordBatch>> ) -> Result<Self>

Create a new in-memory table from the provided schema and record batches

pub async fn load( t: Arc<dyn TableProvider>, output_partitions: Option<usize>, state: &SessionState ) -> Result<Self>

Create a mem table by reading from another data source

Trait Implementations

impl Debug for MemTable

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TableProvider for MemTable

fn insert_into<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 SessionState, input: Arc<dyn ExecutionPlan> ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Returns an ExecutionPlan that inserts the execution results of a given ExecutionPlan into this MemTable.

The ExecutionPlan must have the same schema as this MemTable.

Arguments

• state - The SessionState containing the context for executing the plan.
• input - The ExecutionPlan to execute and insert.

Returns

• A plan that returns the number of rows written.

fn as_any(&self) -> &dyn Any

Returns the table provider as Any so that it can be downcast to a specific implementation.

fn schema(&self) -> SchemaRef

Get a reference to the schema for this table

fn table_type(&self) -> TableType

Get the type of this table for metadata/catalog purposes.

fn scan<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, _state: &'life1 SessionState, projection: Option<&'life2 Vec<usize>>, _filters: &'life3 [Expr], _limit: Option<usize> ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Create an ExecutionPlan that will scan the table. The table provider will be usually responsible of grouping the source data into partitions that can be efficiently parallelized or distributed.

fn get_table_definition(&self) -> Option<&str>

Get the create statement used to create this table, if available.

fn get_logical_plan(&self) -> Option<&LogicalPlan>

Get the Logical Plan of this table, if available.

fn supports_filter_pushdown( &self, _filter: &Expr ) -> Result<TableProviderFilterPushDown>

👎Deprecated since 20.0.0: use supports_filters_pushdown instead

Tests whether the table provider can make use of a filter expression to optimise data retrieval.

fn supports_filters_pushdown( &self, filters: &[&Expr] ) -> Result<Vec<TableProviderFilterPushDown>>

Tests whether the table provider can make use of any or all filter expressions to optimise data retrieval.

fn statistics(&self) -> Option<Statistics>

Get statistics for this table, if available