Module python_udf 

Distributed Python UDF execution.

The engine’s executors are pure Rust with no embedded interpreter, so a Python-callable UDF cannot run in-process there. This module runs it in a persistent python3 worker subprocess instead (the model PySpark uses): the client cloudpickles the callable and ships the bytes with the query; the executor spawns one worker per engine and applies the UDF to each Arrow batch over a length-framed stdin/stdout protocol. The worker caches each UDF by id after first use, so the pickle travels once.

Requires python3 on PATH with pyarrow and cloudpickle (plus whatever the UDF itself imports) available in the runtime environment.

Structs

PythonWorkerAggregateUdf

An aggregate UDF whose implementation is a cloudpickled Python callable.

PythonWorkerPool

A persistent python3 worker that applies cloudpickled UDFs over Arrow IPC. One pool is shared by every Python UDF in an engine; access is serialized through the mutex (one in-flight batch at a time per worker).

PythonWorkerUdf

A scalar UDF whose implementation is a cloudpickled Python callable executed in a PythonWorkerPool. Ships to and runs on the distributed executors.

Functions

global_pool

Process-global worker pool. One python3 worker per process (executor or embedded engine) is spawned lazily on first Python-UDF use and shared by all engines/tasks; UDFs are distinguished by name, and access is serialized. This avoids one process-spawn per UDF and keeps hot imports (numpy, a model) loaded.