Module materialized 

Code for incremental view maintenance against Hive-partitioned tables.

An example of a Hive-partitioned table is the ListingTable. By analyzing the fragment of the materialized view query pertaining to the partition columns, we can derive a build graph that relates the files of a materialized views and the files of the tables it depends on.

Two central traits are defined:

• ListingTableLike: a trait that abstracts Hive-partitioned tables in object storage;
• Materialized: a materialized ListingTableLike defined by a user-provided query.

Note that all implementations of ListingTableLike and Materialized must be registered using the register_listing_table and register_materialized functions respectively, otherwise the tables may not be detected by the incremental view maintenance code, including components such as FileMetadata, RowMetadataRegistry, or the mv_dependencies UDTF.

By default, ListingTableLike is implemented for ListingTable,

Modules

dependencies

Track dependencies of materialized data in object storage

file_metadata

A virtual table that exposes files in object storage.

row_metadata

Pluggable metadata sources for incremental view maintenance

Constants

META_COLUMN

The identifier of the column that RowMetadataSource implementations should store row metadata in.

Traits

ListingTableLike

A TableProvider whose data is backed by Hive-partitioned files in object storage.

Materialized

A hive-partitioned table in object storage that is defined by a user-provided query.

Functions

cast_to_listing_table

Attempt to cast the given TableProvider into a ListingTableLike. If the table’s type has not been registered using register_listing_table, will return None.

cast_to_materialized

Attempt to cast the given TableProvider into a Materialized. If the table’s type has not been registered using register_materialized, will return None.

register_listing_table

Register a ListingTableLike implementation in this registry. This allows cast_to_listing_table to easily downcast a TableProvider into a ListingTableLike where possible.

register_materialized

Register a Materialized implementation in this registry. This allows cast_to_materialized to easily downcast a TableProvider into a Materialized where possible.