Struct MmapDataset 

pub struct MmapDataset { /* private fields */ }

A memory-mapped dataset backed by a Parquet file.

This dataset type memory-maps the underlying file, allowing efficient access to large datasets without loading everything into memory. The OS handles paging data in and out as needed.

Performance Characteristics

• Memory efficient: Only pages accessed are loaded into RAM
• Fast startup: No need to read entire file upfront
• Random access: Efficient access to any row
• OS-managed caching: Leverages OS page cache

Limitations

• File must remain accessible during dataset lifetime
• Not available on WASM targets
• Requires file to be on a seekable filesystem

Implementations

impl MmapDataset

pub fn open(path: impl AsRef<Path>) -> Result<Self>

Opens a Parquet file as a memory-mapped dataset.

Arguments

• path - Path to the Parquet file

Errors

Returns an error if:

• The file cannot be opened
• The file is not valid Parquet
• Memory mapping fails
• The file is empty

Example

use alimentar::MmapDataset;

let dataset = MmapDataset::open("data.parquet").unwrap();

pub fn open_with_batch_size( path: impl AsRef<Path>, batch_size: usize, ) -> Result<Self>

Opens a Parquet file with a specified batch size.

Arguments

• path - Path to the Parquet file
• batch_size - Number of rows per batch

Errors

Returns an error if opening or parsing fails.

pub fn path(&self) -> &Path

Returns the path to the underlying file.

pub fn mmap_size(&self) -> usize

Returns the size of the memory-mapped region in bytes.

pub fn to_arrow_dataset(&self) -> Result<ArrowDataset>

Converts this memory-mapped dataset to an in-memory ArrowDataset.

This is useful when you need to modify the data or when you want to ensure all data is in memory for faster access.

Errors

Returns an error if the conversion fails.

impl MmapDataset

pub fn try_clone(&self) -> Result<Self>

Try to clone this dataset by re-opening the underlying file.

This can fail if the file has been deleted or is no longer accessible.

Trait Implementations

impl Dataset for MmapDataset

fn len(&self) -> usize

Returns the total number of rows in the dataset.

fn get(&self, index: usize) -> Option<RecordBatch>

Returns a single row as a RecordBatch with one row.

fn schema(&self) -> SchemaRef

Returns the schema of the dataset.

fn iter(&self) -> Box<dyn Iterator<Item = RecordBatch> + Send + '_>

Returns an iterator over all RecordBatches in the dataset.

fn num_batches(&self) -> usize

Returns the number of batches in the dataset.

fn get_batch(&self, index: usize) -> Option<&RecordBatch>

Returns a specific batch by index.

fn is_empty(&self) -> bool

Returns true if the dataset contains no rows.

impl Debug for MmapDataset

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.