Module streaming 

Streaming dataset format with lazy chunk loading

Enables lazy loading of large datasets (>100MB) by reading only the header and chunk index initially, then loading chunks on demand.

Example

use alimentar::format::streaming::StreamingDataset;

// Open dataset (only reads header + index)
let dataset = StreamingDataset::open("large_data.ald")?;

// Access chunks lazily
println!("Total rows: {}", dataset.num_rows());
for chunk in dataset.chunks() {
    println!("Chunk with {} rows", chunk?.num_rows());
}

Structs

ChunkEntry

Entry in the chunk index describing one chunk’s location

ChunkIndex

Index of all chunks in a streaming dataset

ChunkIterator

Iterator over chunks in a streaming dataset

StreamingDataset

Streaming dataset with lazy chunk loading

Constants

DEFAULT_CHUNK_SIZE

Default chunk size in rows for streaming format

STREAMING_THRESHOLD

Minimum dataset size to recommend streaming (100MB)

Functions

save_streaming

Save a dataset in streaming format