Struct lance_encoding::decoder::DecodeBatchScheduler

pub struct DecodeBatchScheduler { /* private fields */ }

The scheduler for decoding batches

Lance decoding is done in two steps, scheduling, and decoding. The scheduling tends to be lightweight and should quickly figure what data is needed from the disk issue the appropriate I/O requests. A decode task is created to eventually decode the data (once it is loaded) and scheduling moves on to scheduling the next page.

Meanwhile, it’s expected that a decode stream will be setup to run at the same time. Decode tasks take the data that is loaded and turn it into Arrow arrays.

This approach allows us to keep our I/O parallelism and CPU parallelism completely separate since those are often two very different values.

Backpressure should be achieved via the I/O service. Requests that are issued will pile up if the decode stream is not polling quickly enough. The crate::EncodingsIo::submit_request function should return a pending future once there are too many I/O requests in flight.

TODO: Implement backpressure

Implementations

impl DecodeBatchScheduler

pub fn new( schema: &Schema, column_infos: &[ColumnInfo], file_buffer_positions: &Vec<u64> ) -> Self

Creates a new decode scheduler with the expected schema and the column metadata of the file.

TODO: How does this work when doing projection? Need to add tests. Can probably take care of this in lance-file by only passing in the appropriate columns with the projected schema.

pub async fn schedule_range( &mut self, range: Range<u64>, sink: UnboundedSender<Box<dyn LogicalPageDecoder>>, scheduler: &Arc<dyn EncodingsIo> ) -> Result<()>

Schedules the load of a range of rows

Arguments

• range - The range of rows to load
• sink - A channel to send the decode tasks
• scheduler An I/O scheduler to issue I/O requests

pub async fn schedule_take( &mut self, indices: &[u32], sink: UnboundedSender<Box<dyn LogicalPageDecoder>>, scheduler: &Arc<dyn EncodingsIo> ) -> Result<()>

Schedules the load of selected rows

Arguments

• indices - The row indices to load (these must be in ascending order!)
• sink - A channel to send the decode tasks
• scheduler An I/O scheduler to issue I/O requests