Struct StreamingCacheTableProvider 

pub struct StreamingCacheTableProvider { /* private fields */ }

A TableProvider that caches streaming results from an inner TableProvider.

This provider executes a TableProvider and caches the results as they stream in. On subsequent scans, it returns cached batches first, then continues with new batches as they arrive.

Caching Behavior

• First scan: Triggers streaming from the TableProvider. Each batch is cached.
• Subsequent scans (while streaming): Returns cached batches immediately, then waits for new batches as they arrive.
• After streaming complete: Returns all cached batches via a MemTable.

Refresh Behavior

When refresh() is called, a new inner cache is created. Old streams continue reading from the old cache until completion, while new scans use the fresh cache.

Pushdown filters

This provider does not support filter pushdown since it caches the full table. If your table supports filter pushdown, you probably should not use this cache.

Implementations

impl StreamingCacheTableProvider

pub fn new( input_table: Arc<dyn TableProvider>, runtime: AsyncRuntimeHandle, ) -> Self

Create a new streaming cache table provider.

The input_table is called once on the first scan to create the inner stream that will be cached.

pub fn refresh(&self)

Invalidate the cache and prepare for a fresh stream on next scan.

Old streams will continue reading from the old cache until completion. New scans will use a fresh cache.

pub fn is_complete(&self) -> bool

Check if the cache is complete (all data received).

pub fn cached_batch_count(&self) -> usize

Get the current number of cached batches.

pub fn state(&self) -> CacheState

Get the current cache state.

Trait Implementations

impl Debug for StreamingCacheTableProvider

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TableProvider for StreamingCacheTableProvider

fn as_any(&self) -> &dyn Any

Returns the table provider as Any so that it can be downcast to a specific implementation.

fn schema(&self) -> SchemaRef

Get a reference to the schema for this table

fn table_type(&self) -> TableType

Get the type of this table for metadata/catalog purposes.

fn scan<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, state: &'life1 dyn Session, projection: Option<&'life2 Vec<usize>>, filters: &'life3 [Expr], limit: Option<usize>, ) -> Pin<Box<dyn Future<Output = DataFusionResult<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Create an ExecutionPlan for scanning the table with optionally specified projection, filter and limit, described below.

fn supports_filters_pushdown( &self, filters: &[&Expr], ) -> DataFusionResult<Vec<TableProviderFilterPushDown>>

Specify if DataFusion should provide filter expressions to the TableProvider to apply during the scan.

fn constraints(&self) -> Option<&Constraints>

Get a reference to the constraints of the table. Returns:

fn get_table_definition(&self) -> Option<&str>

Get the create statement used to create this table, if available.

fn get_logical_plan(&self) -> Option<Cow<'_, LogicalPlan>>

Get the LogicalPlan of this table, if available.

fn get_column_default(&self, _column: &str) -> Option<&Expr>

Get the default value for a column, if available.

fn scan_with_args<'a, 'life0, 'life1, 'async_trait>( &'life0 self, state: &'life1 dyn Session, args: ScanArgs<'a>, ) -> Pin<Box<dyn Future<Output = Result<ScanResult, DataFusionError>> + Send + 'async_trait>>

where 'a: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Create an ExecutionPlan for scanning the table using structured arguments.

fn statistics(&self) -> Option<Statistics>

Get statistics for this table, if available Although not presently used in mainline DataFusion, this allows implementation specific behavior for downstream repositories, in conjunction with specialized optimizer rules to perform operations such as re-ordering of joins.

fn insert_into<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _input: Arc<dyn ExecutionPlan>, _insert_op: InsertOp, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Return an ExecutionPlan to insert data into this table, if supported.

fn delete_from<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _filters: Vec<Expr>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Delete rows matching the filter predicates.

fn update<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _assignments: Vec<(String, Expr)>, _filters: Vec<Expr>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Update rows matching the filter predicates.

fn truncate<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Remove all rows from the table.