datafusion_materialized_views::materialized::row_metadata

Trait RowMetadataSource

Source 
pub trait RowMetadataSource: Send + Sync {
    // Required methods
    fn name(&self) -> &str;
    fn row_metadata(
        &self,
        table: ResolvedTableReference,
        scan: &TableScan,
    ) -> Result<LogicalPlanBuilder>;
}
Expand description

A source for “row metadata”, that associates rows from a table with metadata used for incremental view maintenance.

Most use cases should default to using FileMetadata for their RowMetadataSource, which uses object store metadata to perform incremental view maintenance on Hive-partitioned tables. However, in some use cases it is necessary to track metadata at a more granular level than Hive partitions. In such cases, users may implement a custom RowMetadataSource containing this metadata.

A RowMetadataSource may contain metadata for multiple tables. As such, it is the user’s responsibility to register each table with the appropriate RowMetadataSource in the RowMetadataRegistry.

Required Methods§

Source

fn name(&self) -> &str

The name of this row metadata source.

Source

fn row_metadata( &self, table: ResolvedTableReference, scan: &TableScan, ) -> Result<LogicalPlanBuilder>

Rewrite this TableScan as query against this RowMetadataSource, this time adding a new struct column, __meta, whose shape conforms to the following schema:

{
    "table_catalog": "string",
    "table_schema": "string",
    "table_name": "string",
    "source_uri": "string",
    "last_modified": "timestamp",
}

The returned data should contain the original table scan, up to multiplicity. That is, for each row in the original table scan, the RowMetadataSource should contain at least one row (but potentially more) with the same values, plus the __meta column.

Implementors§