Trait ShufflePartitionReader 

pub trait ShufflePartitionReader:
    Debug
    + Send
    + Sync {
    // Required method
    fn open_partition(
        &self,
        upstream_stage_index: usize,
        map_task_index: usize,
        partition: usize,
    ) -> BoxFuture<'static, Result<ShuffleFragmentStream, String>>;
}

Executor-side access to upstream shuffle partitions.

The executor implements this over its shuffle store (local reads) and the Flight endpoints delivered with the task assignment (remote reads); krishiv-sql stays free of shuffle/transport dependencies.

Required Methods

fn open_partition( &self, upstream_stage_index: usize, map_task_index: usize, partition: usize, ) -> BoxFuture<'static, Result<ShuffleFragmentStream, String>>

Open one map task’s output for partition of upstream_stage_index.

The returned future resolves once the fragment has been located — a missing partition is an error here, before any rows are produced — and the stream then yields its batches as they are decoded.

Why this streams

This used to return Vec<RecordBatch>: a reduce task materialised each upstream fragment whole, and neither the Flight decode buffers nor the resulting batches passed through the DataFusion memory pool. With three task slots and several fragments in flight per slot, that is hundreds of megabytes the pool cannot see and therefore cannot make anyone spill for — which is how an executor with a 2.6 GB pool reached 4.5 GiB of heap and was OOM-killed on TPC-H q10. Streaming makes a reduce task’s fragment cost one batch instead of one fragment.

A missing fragment (the map task produced no rows for this partition) yields an empty stream, not an error.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors