Module codec 

Python-aware extension codecs.

Datafusion-python plans can carry references to Python-defined objects that the upstream protobuf codecs do not know how to serialize: pure-Python scalar / aggregate / window UDFs, Python query-planning extensions, and so on. Their state lives inside Py<PyAny> callables and closures rather than being recoverable from a name in the receiver’s function registry. To ship a plan across a process boundary (pickle, multiprocessing, Ray actor, datafusion-distributed, etc.) those payloads have to be encoded into the proto wire format itself.

PythonLogicalCodec is the LogicalExtensionCodec that datafusion-python parks on every SessionContext. It wraps a user-supplied (or default) inner codec and adds Python-aware in-band encoding on top: when the encoder sees a Python-defined UDF, the codec cloudpickles the callable + signature into the fun_definition proto field; when the decoder sees a payload it produced, it reconstructs the UDF from the bytes alone — no pre-registration on the receiver. UDFs the codec does not recognise are delegated to inner, which is typically DefaultLogicalExtensionCodec but may be a downstream-supplied FFI codec installed via SessionContext.with_logical_extension_codec(...).

PythonPhysicalCodec is the symmetric wrapper around PhysicalExtensionCodec. Logical and physical layers each have a try_encode_udf / try_decode_udf pair, so a ScalarUDF referenced inside a LogicalPlan, an ExecutionPlan, or a PhysicalExpr must encode identically through either layer for plans to survive a serialization round-trip. Both codecs share the same payload framing for that reason.

Payloads emitted by these codecs are framed as <family_magic: 7 bytes> <version: u8> <py_major: u8> <py_minor: u8> <cloudpickle blob>. The family magic identifies the UDF flavor; the version byte lets the decoder reject too-new or too-old payloads with a clean error instead of falling into an opaque cloudpickle tuple-unpack failure when the tuple shape changes; the Python (major, minor) bytes catch the cloudpickle-cross-minor-version case and raise an actionable error instead of an opaque marshal failure on load (cloudpickle payloads are not portable across Python minor versions). Dispatch precedence on decode: family match + supported version + matching Python version → inner codec → caller’s FunctionRegistry fallback.

Wire-format family registry

Layer + kind	Family prefix
PythonLogicalCodec scalar	DFPYUDF
PythonLogicalCodec agg	DFPYUDA
PythonLogicalCodec window	DFPYUDW
PythonPhysicalCodec scalar	DFPYUDF
PythonPhysicalCodec agg	DFPYUDA
PythonPhysicalCodec window	DFPYUDW
User FFI extension codec	user-chosen
Default codec	(none)

Current wire-format version is [WIRE_VERSION_CURRENT]; supported receive range is WIRE_VERSION_MIN_SUPPORTED..=WIRE_VERSION_CURRENT. Bump [WIRE_VERSION_CURRENT] whenever the cloudpickle tuple shape changes; raise [WIRE_VERSION_MIN_SUPPORTED] when dropping support for an older shape.

Downstream FFI codecs should pick non-colliding family prefixes (use a DF namespace plus a crate-specific suffix). The codec implementations in this module currently delegate every method to inner; the encoder/decoder hooks for each kind are added as the corresponding Python-side type becomes serializable.

Structs

PythonLogicalCodec

LogicalExtensionCodec parked on every SessionContext. Holds the Python-aware encoding hooks for logical-layer types (LogicalPlan, Expr) and delegates everything it does not handle to the composable inner codec — typically DefaultLogicalExtensionCodec, or a downstream FFI codec installed via SessionContext.with_logical_extension_codec(...).

PythonPhysicalCodec

PhysicalExtensionCodec mirror of PythonLogicalCodec parked on the same SessionContext. Carries the Python-aware encoding hooks for physical-layer types (ExecutionPlan, PhysicalExpr) and delegates the rest to inner.