Struct GroupedData 

pub struct GroupedData { /* private fields */ }

Grouped data for performing aggregations.

Mirrors pyspark.sql.GroupedData.

Implementations

impl GroupedData

pub fn pivot( &self, pivot_col: Column, values: Option<Vec<Value>>, ) -> GroupedData

Pivot on a column, optionally with explicit values (GroupedData.pivot).

Mirrors df.groupBy(...).pivot(col, values). When values is None the server computes the distinct values; when supplied they are serialized as pivot-value literals (previously they were dropped entirely).

pub fn input_columns(&self) -> Result<Vec<String>>

Column names of the underlying (pre-grouping) DataFrame. These are passed as the map/apply UDF’s argument columns so the worker’s pandas/Arrow input is named.

pub fn apply_in_pandas( &self, func: CommonInlineUserDefinedFunctionExpression, ) -> DataFrame

Apply a pandas UDF to each group (GroupedData.applyInPandas).

func is built on the Python side (cloudpickled, eval type SQL_GROUPED_MAP_PANDAS_UDF).

pub fn apply_in_arrow( &self, func: CommonInlineUserDefinedFunctionExpression, ) -> DataFrame

Apply an Arrow UDF to each group (GroupedData.applyInArrow).

pub fn apply_in_pandas_with_state( &self, func: CommonInlineUserDefinedFunctionExpression, state_schema: DataType, output_mode: &str, timeout_conf: &str, ) -> DataFrame

Apply a stateful pandas UDF to each group (GroupedData.applyInPandasWithState).

func (built on the Python side with eval type SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE and carrying the output schema as its return type) is combined with the state schema, output mode, and timeout.

pub fn transform_with_state( &self, func: CommonInlineUserDefinedFunctionExpression, output_mode: &str, time_mode: &str, event_time_column_name: Option<&str>, initial_state: Option<&GroupedData>, ) -> DataFrame

GroupedData.transformWithState (row output). func is the cloudpickled stateful processor (eval type in the 211-214 range).

pub fn transform_with_state_in_pandas( &self, func: CommonInlineUserDefinedFunctionExpression, output_schema: DataType, output_mode: &str, time_mode: &str, event_time_column_name: Option<&str>, initial_state: Option<&GroupedData>, ) -> DataFrame

GroupedData.transformWithStateInPandas (carries the output schema in the TransformWithStateInfo).

pub fn cogroup(&self, other: &GroupedData) -> CoGroupedData

Cogroup this grouped data with another (GroupedData.cogroup).

pub fn agg(&self, expressions: Vec<Expression>) -> DataFrame

Perform an aggregation.

pub fn count(&self) -> DataFrame

Count rows in each group.

pub fn sum(&self, columns: Vec<&str>) -> DataFrame

Sum values in each group.

pub fn avg(&self, columns: Vec<&str>) -> DataFrame

Average values in each group.

pub fn min(&self, columns: Vec<&str>) -> DataFrame

Minimum values in each group.

pub fn max(&self, columns: Vec<&str>) -> DataFrame

Maximum values in each group.

pub fn mean(&self, columns: Vec<&str>) -> DataFrame

Mean (alias for avg).

Trait Implementations

impl Clone for GroupedData

fn clone(&self) -> GroupedData

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.