Module materialized 

Code for incremental view maintenance against Hive-partitioned tables.

An example of a Hive-partitioned table is the ListingTable. By analyzing the fragment of the materialized view query pertaining to the partition columns, we can derive a build graph that relates the files of a materialized views and the files of the tables it depends on.

Two central traits are defined:

• ListingTableLike: a trait that abstracts Hive-partitioned tables in object storage;
• Materialized: a materialized ListingTableLike defined by a user-provided query.

Note that all implementations of ListingTableLike and Materialized must be registered using the register_listing_table and register_materialized functions respectively, otherwise the tables may not be detected by the incremental view maintenance code, including components such as FileMetadata, RowMetadataRegistry, or the mv_dependencies UDTF.

By default, ListingTableLike is implemented for ListingTable,

Modules

dependencies

This module implements a dependency analysis algorithm for materialized views, heavily based on the ListingTableLike trait. Note that materialized views may depend on tables that are not ListingTableLike, as long as they have custom metadata explicitly installed into the RowMetadataRegistry. However, materialized views themself must implement ListingTableLike, as is implied by the type bound Materialized: ListingTableLike.

file_metadata

A virtual table that exposes files in object storage.

row_metadata

Pluggable metadata sources for incremental view maintenance

Constants

META_COLUMN

The identifier of the column that RowMetadataSource implementations should store row metadata in.

Traits

Decorator

A TableProvider that decorates other TableProviders. Sometimes users may implement a TableProvider that overrides functionality of a base TableProvider. This API allows the decorator to also be recognized as ListingTableLike or Materialized automatically.

ListingTableLike

A TableProvider whose data is backed by Hive-partitioned files in object storage.

Materialized

A hive-partitioned table in object storage that is defined by a user-provided query.

Functions

cast_to_listing_table

Attempt to cast the given TableProvider into a ListingTableLike. If the table’s type has not been registered using register_listing_table, will return None.

cast_to_materialized

Attempt to cast the given TableProvider into a Materialized. If the table’s type has not been registered using register_materialized, will return Ok(None).

register_decorator

Register T as a Decorator.

register_listing_table

Register a ListingTableLike implementation in this registry. This allows cast_to_listing_table to easily downcast a TableProvider into a ListingTableLike where possible.

register_materialized

Register a Materialized implementation in this registry. This allows cast_to_materialized to easily downcast a TableProvider into a Materialized where possible.