Struct AsyncPrefetchDataset 

pub struct AsyncPrefetchDataset { /* private fields */ }

A streaming dataset with async prefetch for parallel I/O.

Unlike StreamingDataset which reads synchronously, AsyncPrefetchDataset spawns a background task that reads batches into a channel, allowing the main thread to process while I/O happens.

Example

use alimentar::async_prefetch::AsyncPrefetchDataset;

#[tokio::main]
async fn main() {
    let dataset = AsyncPrefetchDataset::from_parquet("data.parquet", 1024, 4)
        .await
        .unwrap();

    while let Some(batch) = dataset.next().await {
        println!("Processing batch with {} rows", batch.num_rows());
    }
}

Implementations

impl AsyncPrefetchDataset

pub fn new(source: Box<dyn DataSource>, prefetch_size: usize) -> Self

Creates a new async prefetch dataset from a data source.

Arguments

• source - The data source to read from
• prefetch_size - Number of batches to buffer ahead

pub fn from_parquet( path: impl AsRef<Path>, batch_size: usize, prefetch_size: usize, ) -> Result<Self>

Creates an async prefetch dataset from a Parquet file.

Arguments

• path - Path to the Parquet file
• batch_size - Number of rows per batch
• prefetch_size - Number of batches to buffer ahead

Errors

Returns an error if the file cannot be opened.

pub fn schema(&self) -> SchemaRef

Returns the schema of the dataset.

pub async fn next(&mut self) -> Option<Result<RecordBatch>>

Receives the next batch asynchronously.

Returns None when the source is exhausted.

pub fn try_next(&mut self) -> Option<Result<RecordBatch>>

Tries to receive a batch without waiting.

Returns None if no batch is available or the source is exhausted.

pub fn buffered_count(&self) -> usize

Returns the number of batches currently buffered.

Trait Implementations

impl Debug for AsyncPrefetchDataset

Available on crate feature tokio-runtime only.

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.