Trait Partition 

pub trait Partition: 'static + Send {
    // Required methods
    fn as_any(&self) -> &dyn Any;
    fn index(&self) -> usize;
    fn row_count(&self) -> Option<Precision<u64>>;
    fn byte_size(&self) -> Option<Precision<u64>>;
    fn execute(self: Box<Self>) -> VortexResult<SendableArrayStream>;

    // Provided method
    fn serialize(&self) -> VortexResult<Option<Vec<u8>>> { ... }
}

A partition represents a unit of work that can be executed to produce a stream of arrays.

Required Methods

fn as_any(&self) -> &dyn Any

Downcast the partition to a concrete type.

fn index(&self) -> usize

Some unique identifier for partition. If you have an instance of a DataSource, the indices of emitted partitions will stay stable for every scan in this DataSource.

fn row_count(&self) -> Option<Precision<u64>>

Returns an estimate of the row count for this partition.

fn byte_size(&self) -> Option<Precision<u64>>

Returns an estimate of the byte size for this partition.

fn execute(self: Box<Self>) -> VortexResult<SendableArrayStream>

Executes the partition, returning an array stream.

This method must be fast. The returned stream should be lazy — all non-trivial work (I/O, decoding, filtering) must be deferred to when the stream is polled. Expensive operations should be spawned onto the runtime to enable parallel execution across threads.

Provided Methods

fn serialize(&self) -> VortexResult<Option<Vec<u8>>>

Serialize this partition for a remote worker.

Implementors