Struct lance_encoding::encodings::logical::list::ListPageScheduler

pub struct ListPageScheduler { /* private fields */ }

A page scheduler for list fields that encodes offsets in one field and items in another

The list scheduler is somewhat unique because it requires indirect I/O. We cannot know the ranges we need simply by looking at the metadata. This means that list scheduling doesn’t fit neatly into the two-thread schedule-loop / decode-loop model. To handle this, when a list page is scheduled, we only schedule the I/O for the offsets and then we immediately launch a new tokio task. This new task waits for the offsets, decodes them, and then schedules the I/O for the items. Keep in mind that list items can be lists themselves. If that is the case then this indirection will continue. The decode task that is returned will only finish waiting when all of the I/O has completed.

Whenever we schedule follow-up I/O like this the priority is based on the top-level row index. This helps ensure that earlier rows get finished completely (including follow up tasks) before we perform I/O for later rows.

TODO: Actually implement the priority system described above

Implementations

impl ListPageScheduler

pub fn new( offsets_scheduler: Box<dyn LogicalPageScheduler>, items_schedulers: Vec<Box<dyn LogicalPageScheduler>>, items_type: DataType, offset_type: DataType, null_offset_adjustment: u64 ) -> Self

Trait Implementations

impl Debug for ListPageScheduler

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl LogicalPageScheduler for ListPageScheduler

fn schedule_ranges( &self, ranges: &[Range<u32>], scheduler: &Arc<dyn EncodingsIo>, sink: &UnboundedSender<Box<dyn LogicalPageDecoder>> ) -> Result<()>

Schedules I/O for the requested portions of the page.

fn num_rows(&self) -> u32

The number of rows covered by this page

fn schedule_take( &self, indices: &[u32], scheduler: &Arc<dyn EncodingsIo>, sink: &UnboundedSender<Box<dyn LogicalPageDecoder>> ) -> Result<()>

Schedules I/O for the requested rows (identified by row offsets from start of page) TODO: implement this using schedule_ranges