Struct MemTable

pub struct MemTable {
    pub batches: Vec<Arc<RwLock<Vec<RecordBatch>>>>,
    pub sort_order: Arc<Mutex<RawMutex, Vec<Vec<Sort>>>>,
    /* private fields */
}

In-memory data source for presenting a Vec<RecordBatch> as a data source that can be queried by DataFusion. This allows data to be pre-loaded into memory and then repeatedly queried without incurring additional file I/O overhead.

Fields

batches: Vec<Arc<RwLock<Vec<RecordBatch>>>>

sort_order: Arc<Mutex<RawMutex, Vec<Vec<Sort>>>>

Optional pre-known sort order(s). Must be SortExprs. inserting data into this table removes the order

Implementations

impl MemTable

pub fn try_new( schema: Arc<Schema>, partitions: Vec<Vec<RecordBatch>>, ) -> Result<MemTable, DataFusionError>

Create a new in-memory table from the provided schema and record batches

pub fn with_constraints(self, constraints: Constraints) -> MemTable

Assign constraints

pub fn with_column_defaults( self, column_defaults: HashMap<String, Expr>, ) -> MemTable

Assign column defaults

pub fn with_sort_order(self, sort_order: Vec<Vec<Sort>>) -> MemTable

Specify an optional pre-known sort order(s). Must be SortExprs.

If the data is not sorted by this order, DataFusion may produce incorrect results.

DataFusion may take advantage of this ordering to omit sorts or use more efficient algorithms.

Note that multiple sort orders are supported, if some are known to be equivalent,

pub async fn load( t: Arc<dyn TableProvider>, output_partitions: Option<usize>, state: &dyn Session, ) -> Result<MemTable, DataFusionError>

Create a mem table by reading from another data source

Trait Implementations

impl Debug for MemTable

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl TableProvider for MemTable

fn insert_into<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, input: Arc<dyn ExecutionPlan>, insert_op: InsertOp, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, MemTable: 'async_trait,

Returns an ExecutionPlan that inserts the execution results of a given ExecutionPlan into this MemTable.

The ExecutionPlan must have the same schema as this MemTable.

Arguments

• state - The SessionState containing the context for executing the plan.
• input - The ExecutionPlan to execute and insert.

Returns

• A plan that returns the number of rows written.

fn as_any(&self) -> &(dyn Any + 'static)

Returns the table provider as Any so that it can be downcast to a specific implementation.

fn schema(&self) -> Arc<Schema>

Get a reference to the schema for this table

fn constraints(&self) -> Option<&Constraints>

Get a reference to the constraints of the table. Returns:

fn table_type(&self) -> TableType

Get the type of this table for metadata/catalog purposes.

fn scan<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, state: &'life1 dyn Session, projection: Option<&'life2 Vec<usize>>, _filters: &'life3 [Expr], _limit: Option<usize>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait, MemTable: 'async_trait,

Create an ExecutionPlan for scanning the table with optionally specified projection, filter and limit, described below.

fn get_column_default(&self, column: &str) -> Option<&Expr>

Get the default value for a column, if available.

fn get_table_definition(&self) -> Option<&str>

Get the create statement used to create this table, if available.

fn get_logical_plan(&self) -> Option<Cow<'_, LogicalPlan>>

Get the LogicalPlan of this table, if available.

fn supports_filters_pushdown( &self, filters: &[&Expr], ) -> Result<Vec<TableProviderFilterPushDown>, DataFusionError>

Specify if DataFusion should provide filter expressions to the TableProvider to apply during the scan.

fn statistics(&self) -> Option<Statistics>

Get statistics for this table, if available Although not presently used in mainline DataFusion, this allows implementation specific behavior for downstream repositories, in conjunction with specialized optimizer rules to perform operations such as re-ordering of joins.