Struct ArrowDataset 

pub struct ArrowDataset { /* private fields */ }

An in-memory dataset backed by Arrow RecordBatches.

This is the primary dataset type for alimentar. It stores data as a collection of RecordBatches and provides efficient access patterns for ML training loops.

Example

use alimentar::{ArrowDataset, Dataset};

// Load from parquet
let dataset = ArrowDataset::from_parquet("data.parquet").unwrap();
println!("Dataset has {} rows", dataset.len());

Implementations

impl ArrowDataset

pub fn new(batches: Vec<RecordBatch>) -> Result<Self>

Creates a new ArrowDataset from a vector of RecordBatches.

Errors

Returns an error if:

• The batches vector is empty
• The batches have inconsistent schemas

pub fn from_batch(batch: RecordBatch) -> Result<Self>

Creates an ArrowDataset from a single RecordBatch.

Errors

Returns an error if the batch is empty.

pub fn from_parquet(path: impl AsRef<Path>) -> Result<Self>

Loads a dataset from a Parquet file.

Errors

Returns an error if:

• The file cannot be opened
• The file is not valid Parquet
• The file is empty

pub fn to_parquet(&self, path: impl AsRef<Path>) -> Result<()>

Saves the dataset to a Parquet file.

Errors

Returns an error if:

• The file cannot be created
• Writing fails

pub fn from_ipc(path: impl AsRef<Path>) -> Result<Self>

Loads a dataset from an Arrow IPC file (Issue #2: Enhanced Data Loading)

Arrow IPC (Inter-Process Communication) format enables zero-copy data sharing. This is the native Arrow file format with optimal read performance.

Arguments

• path - Path to the Arrow IPC file (typically .arrow or .ipc extension)

Errors

Returns an error if:

• The file cannot be opened
• The file is not valid Arrow IPC format
• The file is empty

Example

let dataset = ArrowDataset::from_ipc("data.arrow").unwrap();

pub fn to_ipc(&self, path: impl AsRef<Path>) -> Result<()>

Saves the dataset to an Arrow IPC file (Issue #2: Enhanced Data Loading)

Creates a file in Arrow IPC format, the native Arrow format. This format provides optimal read performance for Arrow-based tools.

Arguments

• path - Path for the output file

Errors

Returns an error if the file cannot be created or writing fails.

Example

dataset.to_ipc("output.arrow").unwrap();

pub fn from_ipc_stream(path: impl AsRef<Path>) -> Result<Self>

Loads a dataset from an Arrow IPC stream file (Issue #2: Enhanced Data Loading)

Arrow IPC streaming format is designed for streaming scenarios where the schema is sent first, followed by record batches. The file extension is typically .arrows.

Arguments

• path - Path to the Arrow IPC stream file

Errors

Returns an error if parsing fails or the file is empty.

pub fn to_ipc_stream(&self, path: impl AsRef<Path>) -> Result<()>

Saves the dataset to an Arrow IPC stream file (Issue #2: Enhanced Data Loading)

Creates a file in Arrow IPC streaming format. This format is suitable for streaming scenarios and produces slightly smaller files than the standard IPC file format.

Arguments

• path - Path for the output file (typically .arrows extension)

Errors

Returns an error if the file cannot be created or writing fails.

pub fn from_csv(path: impl AsRef<Path>) -> Result<Self>

Loads a dataset from a CSV file.

Arguments

• path - Path to the CSV file

Errors

Returns an error if:

• The file cannot be opened
• The file is not valid CSV
• The file is empty

pub fn from_csv_with_options( path: impl AsRef<Path>, options: CsvOptions, ) -> Result<Self>

Loads a dataset from a CSV file with options.

Arguments

• path - Path to the CSV file
• options - CSV parsing options

Errors

Returns an error if parsing fails or the file is empty.

pub fn to_csv(&self, path: impl AsRef<Path>) -> Result<()>

Saves the dataset to a CSV file.

Errors

Returns an error if the file cannot be created or writing fails.

pub fn from_json(path: impl AsRef<Path>) -> Result<Self>

Loads a dataset from a JSON Lines (JSONL) file.

Each line in the file should be a valid JSON object representing a row.

Errors

Returns an error if the file cannot be opened or parsed.

pub fn from_json_with_options( path: impl AsRef<Path>, options: JsonOptions, ) -> Result<Self>

Loads a dataset from a JSON Lines file with options.

Errors

Returns an error if parsing fails or the file is empty.

pub fn to_json(&self, path: impl AsRef<Path>) -> Result<()>

Saves the dataset to a JSON Lines (JSONL) file.

Each row is written as a single JSON object on its own line.

Errors

Returns an error if the file cannot be created or writing fails.

pub fn from_parquet_bytes(data: &[u8]) -> Result<Self>

Loads a dataset from Parquet bytes in memory.

Errors

Returns an error if the data is not valid Parquet.

pub fn to_parquet_bytes(&self) -> Result<Vec<u8>>

Converts the dataset to Parquet bytes.

Errors

Returns an error if serialization fails.

pub fn from_csv_str(data: &str) -> Result<Self>

Loads a dataset from a CSV string.

Errors

Returns an error if the string is not valid CSV.

pub fn from_json_str(data: &str) -> Result<Self>

Loads a dataset from a JSON string.

Errors

Returns an error if the string is not valid JSON.

pub fn batches(&self) -> &[RecordBatch]

Returns the underlying batches.

pub fn into_batches(self) -> Vec<RecordBatch>

Consumes the dataset and returns the underlying batches.

pub fn with_transform<T: Transform>(&self, transform: &T) -> Result<Self>

Applies a transform to create a new dataset.

Errors

Returns an error if the transform fails on any batch.

pub fn rows(&self) -> RowIterator<'_>

Returns an iterator over rows as single-row RecordBatches.

Trait Implementations

impl Clone for ArrowDataset

fn clone(&self) -> ArrowDataset

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Dataset for ArrowDataset

fn len(&self) -> usize

Returns the total number of rows in the dataset.

fn get(&self, index: usize) -> Option<RecordBatch>

Returns a single row as a RecordBatch with one row.

fn schema(&self) -> SchemaRef

Returns the schema of the dataset.

fn iter(&self) -> Box<dyn Iterator<Item = RecordBatch> + Send + '_>

Returns an iterator over all RecordBatches in the dataset.

fn num_batches(&self) -> usize

Returns the number of batches in the dataset.

fn get_batch(&self, index: usize) -> Option<&RecordBatch>

Returns a specific batch by index.

fn is_empty(&self) -> bool

Returns true if the dataset contains no rows.

impl Debug for ArrowDataset

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.