pub struct BoundedExecutor { /* private fields */ }Expand description
Parse an input in bounded batches to stay within a memory budget.
Implementations§
Source§impl BoundedExecutor
impl BoundedExecutor
pub fn new(budget: MemoryBudget) -> Self
Sourcepub fn with_split_cap(self, cap: usize) -> Self
pub fn with_split_cap(self, cap: usize) -> Self
Override the split cap (default MAX_SPLIT_CHUNKS). Raising it may be
needed for very large files with small budgets, where capped batches
would otherwise overshoot the budget.
Sourcepub fn run_bytes_stream<P, C>(
&self,
bytes: &[u8],
splitter: &dyn Splitter,
parser: P,
plan: Arc<ExecutionPlan>,
consumer: &mut C,
) -> Result<()>
pub fn run_bytes_stream<P, C>( &self, bytes: &[u8], splitter: &dyn Splitter, parser: P, plan: Arc<ExecutionPlan>, consumer: &mut C, ) -> Result<()>
Parse an in-memory byte slice in bounded batches, calling consumer per batch.
Chunks are sliced directly from bytes; no file I/O occurs. This is
the streaming entry point for adapters holding decompressed data.
The budget controls batch sizing; allocator, parser, input, and Arrow
overhead can raise process RSS above this target.
Sourcepub fn run_bytes_stream_with_stats<P, C>(
&self,
bytes: &[u8],
splitter: &dyn Splitter,
parser: P,
plan: Arc<ExecutionPlan>,
consumer: &mut C,
) -> Result<StreamStats>
pub fn run_bytes_stream_with_stats<P, C>( &self, bytes: &[u8], splitter: &dyn Splitter, parser: P, plan: Arc<ExecutionPlan>, consumer: &mut C, ) -> Result<StreamStats>
Like BoundedExecutor::run_bytes_stream, but reports a
StreamStats summary including the peak tracked engine memory.
The input slice is caller-owned, so it is not charged to the ledger;
peak_tracked_bytes covers builders and in-flight batches only.
Sourcepub fn run_bytes<P>(
&self,
bytes: &[u8],
splitter: &dyn Splitter,
parser: P,
plan: Arc<ExecutionPlan>,
) -> Result<Vec<RecordBatch>>
pub fn run_bytes<P>( &self, bytes: &[u8], splitter: &dyn Splitter, parser: P, plan: Arc<ExecutionPlan>, ) -> Result<Vec<RecordBatch>>
Parse an in-memory byte slice in bounded batches, returning one
RecordBatch per batch.
Chunks are sliced directly from bytes; no file I/O occurs. This is
the entry point for adapters holding decompressed or streamed-in data.
Sourcepub fn run_stream<P, C>(
&self,
path: &Path,
splitter: &dyn Splitter,
parser: P,
plan: Arc<ExecutionPlan>,
prefault: bool,
consumer: &mut C,
) -> Result<()>
pub fn run_stream<P, C>( &self, path: &Path, splitter: &dyn Splitter, parser: P, plan: Arc<ExecutionPlan>, prefault: bool, consumer: &mut C, ) -> Result<()>
Parse path in batches, calling consumer per batch (streaming).
Compressed inputs are decompressed up front and served via
run_bytes_stream. Uncompressed inputs with mmap use the seek-based
path with a reusable chunk buffer to keep RSS constant.
Sourcepub fn run_stream_with_stats<P, C>(
&self,
path: &Path,
splitter: &dyn Splitter,
parser: P,
plan: Arc<ExecutionPlan>,
prefault: bool,
consumer: &mut C,
) -> Result<StreamStats>
pub fn run_stream_with_stats<P, C>( &self, path: &Path, splitter: &dyn Splitter, parser: P, plan: Arc<ExecutionPlan>, prefault: bool, consumer: &mut C, ) -> Result<StreamStats>
Like BoundedExecutor::run_stream, but reports a StreamStats
summary including the peak tracked engine memory.
Sourcepub fn run<P>(
&self,
path: &Path,
splitter: &dyn Splitter,
parser: P,
plan: Arc<ExecutionPlan>,
prefault: bool,
) -> Result<Vec<RecordBatch>>
pub fn run<P>( &self, path: &Path, splitter: &dyn Splitter, parser: P, plan: Arc<ExecutionPlan>, prefault: bool, ) -> Result<Vec<RecordBatch>>
Parse path in batches, returning one RecordBatch per batch.
The caller is responsible for concatenating the batches if a single table is desired.
Compressed inputs (see InputBuffer::open) are decompressed up
front and served from memory via BoundedExecutor::run_bytes.
Uncompressed inputs on platforms with the mmap feature retain the
seek-based path: the mapping is dropped after planning and each chunk
is read from disk on demand, keeping resident memory low.
Auto Trait Implementations§
impl Freeze for BoundedExecutor
impl RefUnwindSafe for BoundedExecutor
impl Send for BoundedExecutor
impl Sync for BoundedExecutor
impl Unpin for BoundedExecutor
impl UnsafeUnpin for BoundedExecutor
impl UnwindSafe for BoundedExecutor
Blanket Implementations§
impl<T> Allocation for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more