Module materialized

Code for incremental view maintenance against Hive-partitioned tables.

An example of a Hive-partitioned table is the ListingTable. By analyzing the fragment of the materialized view query pertaining to the partition columns, we can derive a build graph that relates the files of a materialized views and the files of the tables it depends on.

Two central traits are defined:

• ListingTableLike: a trait that abstracts Hive-partitioned tables in object storage;
• Materialized: a materialized ListingTableLike defined by a user-provided query.

Note that all implementations of ListingTableLike and Materialized must be registered using the register_listing_table and register_materialized functions respectively, otherwise the tables may not be detected by the incremental view maintenance code, including components such as FileMetadata, RowMetadataRegistry, or the mv_dependencies UDTF.

By default, ListingTableLike is implemented for ListingTable,

Modules

• dependencies
  Track dependencies of materialized data in object storage
• file_metadata
  A virtual table that exposes files in object storage.
• row_metadata
  Pluggable metadata sources for incremental view maintenance

Constants

• META_COLUMN
  The identifier of the column that RowMetadataSource implementations should store row metadata in.

Traits

• ListingTableLike
  A TableProvider whose data is backed by Hive-partitioned files in object storage.
• Materialized
  A hive-partitioned table in object storage that is defined by a user-provided query.

Functions

• cast_to_listing_table
  Attempt to cast the given TableProvider into a ListingTableLike. If the table’s type has not been registered using register_listing_table, will return None.
• cast_to_materialized
  Attempt to cast the given TableProvider into a Materialized. If the table’s type has not been registered using register_materialized, will return None.
• register_listing_table
  Register a ListingTableLike implementation in this registry. This allows cast_to_listing_table to easily downcast a TableProvider into a ListingTableLike where possible.
• register_materialized
  Register a Materialized implementation in this registry. This allows cast_to_materialized to easily downcast a TableProvider into a Materialized where possible.