Skip to main content

vortex_array/arrays/widen/
mod.rs

1// SPDX-License-Identifier: Apache-2.0
2// SPDX-FileCopyrightText: Copyright the Vortex contributors
3
4//! An in-memory encoding that stores integer and decimal arrays at a narrower canonical width.
5//!
6//! A [`WidenArray`] keeps the logical dtype of an integer or decimal array, and stores its values
7//! and their validity in a child with the same nullability and a strictly narrower canonical width.
8//!
9//! An integer child has a narrower integer type of the same signedness. For example, an `i64` array
10//! whose values are all between 0 and 200 can store `i16` values, but not `u8` values.
11//! [`WidenArray::encode`] picks the narrowest such type for a [`PrimitiveArray`].
12//!
13//! A decimal child has the logical scale and a lower precision, and **must** be valid for its dtype.
14//! Comparisons, bounds checks, and fills use the child dtype to bound its valid values.
15//! [`WidenArray::encode_decimal`] and [`WidenArray::from_stored_decimal`] choose child precision
16//! from valid values or supplied bounds, at the logical scale. A widening decimal cast keeps the
17//! source dtype on its child. Decimal execution copies child values directly into the logical
18//! canonical width, since yielding a cast would produce another `Widen` instead of canonical values.
19//!
20//! Narrowing is a physical choice, so it **must not** change a result. Execution casts the values
21//! to the logical dtype for integers, and an operation runs on the values only where it gives the
22//! canonical result. A `Widen` never wraps another `Widen`.
23//!
24//! Take executes its gather instead of leaving a lazy [`Dict`] under a `Widen`, because code such
25//! as the DuckDB exporter and the dict layout writer looks for a `Dict` at the root of an array.
26//!
27//! `Widen` has no wire form yet. The default [`ArraySession`] registers [`WidenPlugin`], which
28//! writes a decimal `Widen` with a [`Decimal`](crate::arrays::Decimal) leaf using the legacy
29//! `vortex.decimal` wire form. The leaf supplies the stored width and the containing format supplies
30//! the parent dtype. Serialization keeps the parent statistics and never executes the array.
31//! Integer arrays and non-decimal leaves require explicit preparation by the caller.
32//!
33//! [`ArraySession`]: crate::session::ArraySession
34//! [`Dict`]: crate::arrays::Dict
35//! [`PrimitiveArray`]: crate::arrays::PrimitiveArray
36
37use vortex_session::VortexSession;
38
39mod array;
40pub use array::WidenArraySlotsExt;
41pub use array::WidenSlots;
42pub use array::WidenSlotsView;
43
44pub(crate) mod compute;
45mod encode;
46
47mod vtable;
48pub use vtable::Widen;
49pub use vtable::WidenArray;
50pub use vtable::WidenPlugin;
51
52use crate::array::VTable;
53use crate::arrays::Dict;
54use crate::arrays::dict::TakeExecuteAdaptor;
55use crate::arrays::widen::compute::numeric::WidenNumericKernel;
56use crate::optimizer::kernels::ArrayKernelsExt;
57use crate::scalar_fn::ScalarFnVTable;
58use crate::scalar_fn::fns::binary::Binary;
59use crate::scalar_fn::fns::binary::CompareExecuteAdaptor;
60use crate::scalar_fn::fns::zip::Zip;
61use crate::scalar_fn::fns::zip::ZipExecuteAdaptor;
62
63pub(crate) fn initialize(session: &VortexSession) {
64    let kernels = session.kernels();
65    kernels.register_execute_parent_kernel(Binary.id(), Widen, CompareExecuteAdaptor(Widen));
66    kernels.register_execute_parent_kernel(Binary.id(), Widen, WidenNumericKernel);
67    kernels.register_execute_parent_kernel(Dict.id(), Widen, TakeExecuteAdaptor(Widen));
68    kernels.register_execute_parent_kernel(Zip.id(), Widen, ZipExecuteAdaptor(Widen));
69}
70
71#[cfg(test)]
72mod tests;