Module operator 

This module defines a new way of modelling arrays and expressions in Vortex. To avoid naming conflicts, we refer to the new model as “operators”.

Operators form a more traditional “logical plan” as might be seen in other query engines. Each operator supports one primary function which is to produce a canonical representation of its data, known as canonicalization. Operators have the option to produce this canonical form using different execution models, including batch, pipelined, and GPU.

Initial designs for this module involved passing masks down through the physical execution tree as futures, allowing operators to skip computation for rows that are not needed. We ultimately decided against this approach and instead introduce a Filter operator that can be pushed down in the same way as any other operator.

On the one hand, this means common subtree elimination is much easier, since we know the mask or identity of the mask future inside the filter operator up-front. On the other hand, it means that an operator no longer has a known length. In the end state, we will redefine a Vortex array to be a wrapped around an operator that does have a known length, amongst other properties (such as non-blocking evaluation).

We also introduce the idea of an executor that can evaluate an operator tree efficiently. It supports common subtree elimination, as well as extracting sub-graphs for pipelined and GPU execution. The executor is also responsible for managing memory and scheduling work across different execution resources.

Modules

canonical

compare

filter

getitem

metrics

slice

Structs

OperatorKey

A wrapper type to implement Hash, PartialEq, and Eq using the semantics defined by OperatorHash and OperatorEq.

Enums

DisplayFormat

Traits

BatchBindCtx

BatchExecution

The primary execution trait for operators.

BatchOperator

The default execution mode for an operator is batch mode.

DynOperatorEq

DynOperatorHash

Operator

An operator represents a node in a logical query plan.

OperatorEq

An equality trait for operators that loosens the semantics to permit pointer-based equality for data objects such as buffers.

OperatorHash

A hash trait for operators that loosens the semantics to permit pointer-based hashing for data objects such as buffers.

Type Aliases

BatchExecutionRef

OperatorId

OperatorRef