Module morsel 

Structures for Morsel Driven IO.

NOTE: As of DataFusion 54.0.0, these are experimental APIs that may change substantially.

Morsel Driven IO is a technique for parallelizing the reading of large files by dividing them into smaller “morsels” that are processed independently.

It is inspired by the paper Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age.

Structs

MorselPlan

Return result of MorselPlanner::plan.

PendingMorselPlanner

Wrapper for I/O that must complete before planning can continue.

Traits

Morsel

A Morsel of work ready to resolve to a stream of RecordBatches.

MorselPlanner

A Morsel Planner is responsible for creating morsels for a given scan.

Morselizer

A Morselizer takes a single PartitionedFile and creates the initial planner for that file.