Struct PythonWorkerAggregateUdf 

pub struct PythonWorkerAggregateUdf { /* private fields */ }

An aggregate UDF whose implementation is a cloudpickled Python callable.

Semantics follow PySpark’s GROUPED_AGG pandas UDF: the accumulated rows of a group are buffered (as Arrow IPC frames appended into AggState) and the callable is applied to the whole group exactly once at finalize. This makes the aggregate trivially mergeable across partitions and executors — merge is byte concatenation of two partial buffers — so it works in distributed two-phase aggregation (partial per map task, final after the shuffle) through the existing crate::udf KrishivAggregateAccumulator bridge.

The callable receives each input column of the group as a numpy array (one positional argument per input column) and returns a Python scalar; a callable marked _krishiv_arrow_udf=True instead receives the whole Arrow batch.

Implementations

impl PythonWorkerAggregateUdf

pub fn new( name: impl Into<String>, pickle: Vec<u8>, input_schema: Schema, output_field: Field, pool: Arc<PythonWorkerPool>, ) -> Self

Trait Implementations

impl AggregateUdf for PythonWorkerAggregateUdf

fn name(&self) -> &str

Unique name used to look up this UDAF in a UdfRegistry.

fn input_schema(&self) -> &Schema

The schema of the input columns this UDAF expects.

fn output_field(&self) -> &Field

The output field produced when the UDAF is finalised.

fn accumulate( &self, state: &mut AggState, batch: &RecordBatch, ) -> Result<(), UdfError>

Merge new data from batch into state.

fn merge(&self, a: AggState, b: AggState) -> Result<AggState, UdfError>

Merge two partial states into one (for distributed execution).

fn finalize(&self, state: AggState) -> Result<ScalarValue, UdfError>

Produce a final result from an accumulated state.

fn volatility(&self) -> Volatility

Volatility classification of the UDAF (S3).

impl Debug for PythonWorkerAggregateUdf

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.