Struct StreamingDataset 

pub struct StreamingDataset { /* private fields */ }

Streaming dataset with lazy chunk loading

Uses file seeking to load chunks on demand, minimizing memory usage.

Implementations

impl StreamingDataset

pub fn open(path: impl AsRef<Path>) -> Result<Self>

Open a streaming dataset from a file

Only reads the header, metadata, and chunk index initially. Chunks are loaded on demand.

Errors

Returns error if file cannot be opened, is not a valid .ald file, or does not have the STREAMING flag set.

pub fn num_rows(&self) -> u64

Get total row count

pub fn num_chunks(&self) -> usize

Get number of chunks

pub fn schema(&self) -> SchemaRef

Get the schema

pub fn get_chunk(&self, chunk_idx: usize) -> Result<RecordBatch>

Load a specific chunk by index

Errors

Returns error if chunk index is out of bounds or decompression fails.

pub fn get_rows(&self, start: u64, count: u64) -> Result<RecordBatch>

Get random access to rows by range

Errors

Returns error if row range is out of bounds.

pub fn chunks(&self) -> ChunkIterator<'_>

Iterate over all chunks

Trait Implementations

impl Debug for StreamingDataset

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.