Struct TableValuedFunction 

pub struct TableValuedFunction { /* private fields */ }

Table-valued functions for Spark SQL.

Mirrors pyspark.sql.connect.tvf.TableValuedFunction.

Implementations

impl TableValuedFunction

pub fn udtf( &self, name: &str, arguments: Vec<Column>, return_type: Option<DataType>, eval_type: i32, command: Vec<u8>, python_ver: String, deterministic: bool, ) -> DataFrame

Invoke a Python user-defined table function (UDTF), returning its output as a DataFrame. Mirrors pyspark.sql.functions.udtf. The Python side cloudpickles the handler and sets eval_type (300 SQL_TABLE_UDF, 301 SQL_ARROW_TABLE_UDF, 302 SQL_ARROW_UDTF); command is the pickled payload.

pub fn range( &self, start: i64, end: Option<i64>, step: i64, num_partitions: Option<i32>, ) -> Result<DataFrame>

Create a DataFrame representing a range of integers.

This is a table-valued function that generates a sequence of integers from start to end with the given step.

Arguments

• start - Starting value (inclusive)
• end - Ending value (exclusive) or the only value if called with one argument
• step - Step size between values (default 1)
• num_partitions - Number of partitions (optional)

Example

let df = spark.tvf().range(0, 100, 1, None)?;

pub fn explode(&self, collection: &Column) -> Result<DataFrame>

Explode an array or map column.

Returns a DataFrame where each array/map element becomes a separate row.

Arguments

• collection - The column to explode (must be an array or map)

Example

let df = spark.tvf().explode(col("my_array"))?;

pub fn explode_outer(&self, collection: &Column) -> Result<DataFrame>

Explode an array or map column, preserving nulls as empty rows.

Similar to explode, but null values are preserved.

pub fn inline(&self, input: &Column) -> Result<DataFrame>

Inline an array of structs column.

Each struct becomes a separate row with its fields as columns.

Arguments

• input - The column to inline (must be an array of structs)

Example

let df = spark.tvf().inline(col("struct_array"))?;

pub fn inline_outer(&self, input: &Column) -> Result<DataFrame>

Inline an array of structs column, preserving nulls as empty rows.

Similar to inline, but null values are preserved.

pub fn json_tuple<C: Into<Column>>( &self, input: &Column, fields: impl IntoIterator<Item = C>, ) -> Result<DataFrame>

JSON tuple function.

Extracts fields from a JSON string.

Arguments

• input - The JSON column
• fields - Field names to extract

Example

let df = spark.tvf().json_tuple(col("json_str"), vec![col("field1"), col("field2")])?;

pub fn posexplode(&self, collection: &Column) -> Result<DataFrame>

Explode an array column with position indices.

Similar to explode, but also includes the position of each element.

pub fn posexplode_outer(&self, collection: &Column) -> Result<DataFrame>

Explode an array column with position indices, preserving nulls.

Similar to posexplode, but null values are preserved.

pub fn stack<C: Into<Column>>( &self, n: &Column, fields: impl IntoIterator<Item = C>, ) -> Result<DataFrame>

Stack function.

Converts multiple columns into rows.

Arguments

• n - The number of columns to stack
• fields - The columns to stack

pub fn collations(&self) -> Result<DataFrame>

Returns a DataFrame of collation names.

pub fn sql_keywords(&self) -> Result<DataFrame>

Returns a DataFrame of SQL keywords.

pub fn variant_explode(&self, input: &Column) -> Result<DataFrame>

Explode a variant column.

Variant columns contain semi-structured data in Spark SQL.

pub fn variant_explode_outer(&self, input: &Column) -> Result<DataFrame>

Explode a variant column, preserving nulls.

pub fn python_worker_logs(&self) -> Result<DataFrame>

Get Python worker logs as a DataFrame.