Skip to main content

Module dtype

Module dtype 

Source
Expand description

Convert between Vortex vortex_array::dtype::DType and Apache Arrow arrow_schema::DataType.

Apache Arrow’s type system includes physical information, which could lead to ambiguities as Vortex treats encodings as separate from logical types.

The conversions in this module are “naive”: every logical type is encoded in its simplest corresponding Arrow type, and Arrow extension types (other than the builtin temporal types and Parquet Variant) are not understood. The authoritative, plugin-aware conversion entry point is ArrowSession, through its to_arrow_schema / to_arrow_datatype / from_arrow_schema / from_arrow_field / from_arrow_datatype methods.

For this reason, it’s recommended to do as much computation as possible within Vortex, and then materialize an Arrow ArrayRef at the very end of the processing chain.