Module spark_sql_ext 

Spark SQL extensions: LATERAL VIEW, TABLESAMPLE, TRANSFORM, DESCRIBE EXTENDED, etc. Spark SQL feature extensions — pre-processors for SQL constructs that DataFusion doesn’t parse natively.

Supported Spark SQL features:

• LATERAL VIEW: SELECT ... FROM t LATERAL VIEW explode(arr) AS col
• LATERAL VIEW OUTER: SELECT ... FROM t LATERAL VIEW OUTER explode(arr) AS col
• TABLESAMPLE: SELECT ... FROM t TABLESAMPLE (10 PERCENT)
• TRANSFORM: SELECT TRANSFORM(...) FROM t
• DESCRIBE TABLE EXTENDED: DESCRIBE TABLE EXTENDED t
• SHOW TABLE PROPERTIES: SHOW TBLPROPERTIES t

Functions

contains_describe_extended

Detects DESCRIBE TABLE EXTENDED in SQL.

contains_lateral_view

Detects LATERAL VIEW in SQL.

contains_show_tblproperties

Detects SHOW TBLPROPERTIES in SQL.

contains_tablesample

Detects TABLESAMPLE in SQL.

contains_transform

Detects TRANSFORM in SQL.

preprocess_spark_sql

Apply all Spark SQL pre-processing rewrites to a SQL string.

rewrite_describe_extended

Rewrites DESCRIBE TABLE EXTENDED <table> to standard DESCRIBE TABLE <table>.

rewrite_lateral_view

Rewrites Spark-style LATERAL VIEW to standard SQL CROSS JOIN LATERAL.

rewrite_show_tblproperties

Rewrites SHOW TBLPROPERTIES <table> to a query against the catalog.

rewrite_tablesample

Rewrites Spark TABLESAMPLE(n PERCENT) to DataFusion-compatible form.

rewrite_transform

Rewrites Spark TRANSFORM(...) to standard SQL.