pub struct ParallelExecutor;Expand description
Parallel executor that splits an input byte slice into chunks, parses each chunk independently, and returns the resulting Arrow record batches.
Implementations§
Source§impl ParallelExecutor
impl ParallelExecutor
Sourcepub fn parse<P>(
bytes: &[u8],
splitter: &dyn Splitter,
parser: P,
plan: Arc<ExecutionPlan>,
num_chunks: usize,
) -> Result<Vec<RecordBatch>>
pub fn parse<P>( bytes: &[u8], splitter: &dyn Splitter, parser: P, plan: Arc<ExecutionPlan>, num_chunks: usize, ) -> Result<Vec<RecordBatch>>
Parse bytes in parallel using splitter and parser.
- Fast path (no
auto_dict): each chunk is exported as its ownRecordBatchand a vector of batches is returned. Row filters (Equal/NotEqual/Compare) are applied during parse per chunk. - Merge path: chunk builders are merged sequentially so that
auto-dictionary upgrades see the full cardinality, then a single
RecordBatchis returned inside the vector.
Auto Trait Implementations§
impl Freeze for ParallelExecutor
impl RefUnwindSafe for ParallelExecutor
impl Send for ParallelExecutor
impl Sync for ParallelExecutor
impl Unpin for ParallelExecutor
impl UnsafeUnpin for ParallelExecutor
impl UnwindSafe for ParallelExecutor
Blanket Implementations§
impl<T> Allocation for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more