Struct CustomDataFrame

pub struct CustomDataFrame {
    pub df: DataFrame,
    pub table_alias: String,
    /* private fields */
}

Helper function to extract partition values from a RecordBatch based on partition columns. Assumes that each RecordBatch has a single unique partition value.

Fields

df: DataFrame

table_alias: String

Implementations

impl CustomDataFrame

pub async fn new<'a>( file_path: &'a str, columns: Vec<(&'a str, &'a str, bool)>, alias: &'a str, ) -> Self

NEW method for loading and schema definition

pub async fn raw_sql( &self, sql: &str, alias: &str, dfs: &[&CustomDataFrame], ) -> ElusionResult<Self>

Execute a raw SQL query involving multiple CustomDataFrame instances and return a new CustomDataFrame with the results.

Arguments

• sql - The raw SQL query string to execute.
• alias - The alias name for the resulting DataFrame.
• additional_dfs - A slice of references to other CustomDataFrame instances to be registered in the context.

Returns

• ElusionResult<Self> - A new CustomDataFrame containing the result of the SQL query.

pub fn load_csv<'a>( file_path: &'a str, schema: Arc<Schema>, alias: &'a str, ) -> BoxFuture<'a, Result<AliasedDataFrame, DataFusionError>>

LOAD function for CSV file type

pub fn load_json<'a>( file_path: &'a str, alias: &'a str, ) -> BoxFuture<'a, Result<AliasedDataFrame, DataFusionError>>

Loads a JSON file into a DataFusion DataFrame.

Arguments

• file_path - The path to the JSON file.
• schema - The Arrow schema defining the DataFrame columns.
• alias - The alias name for the table within DataFusion.

Returns

pub fn load<'a>( file_path: &'a str, schema: Arc<Schema>, alias: &'a str, ) -> BoxFuture<'a, Result<AliasedDataFrame, DataFusionError>>

unified load() funciton

pub fn aggregation(self, aggregations: Vec<AggregationBuilder>) -> Self

AGGREAGATION helper

pub fn from_subquery(self, sub_df: CustomDataFrame, alias: &str) -> Self

FROM SUBQUERY clause

pub fn with_cte(self, name: &str, cte_df: CustomDataFrame) -> Self

WITH CTE claUse

pub fn union(self, other: CustomDataFrame, all: bool) -> Self

UNION clause

pub fn intersect(self, other: CustomDataFrame, all: bool) -> Self

INTERSECT cluase

pub fn except(self, other: CustomDataFrame, all: bool) -> Self

EXCEPT clause

pub fn select(self, columns: Vec<&str>) -> Self

SELECT clause

pub fn group_by(self, group_columns: Vec<&str>) -> Self

GROUP BY clause

pub fn order_by(self, columns: Vec<&str>, ascending: Vec<bool>) -> Self

ORDER BY clause

pub fn limit(self, count: usize) -> Self

LIMIT lcause

pub fn filter(self, condition: &str) -> Self

FILTER clause Applies a WHERE filter with automatic lowercasing

pub fn having(self, condition: &str) -> Self

Applies a HAVING filter with automatic lowercasing

pub fn join( self, other: CustomDataFrame, condition: &str, join_type: &str, ) -> Self

JOIN clause

pub fn window( self, func: &str, column: &str, partition_by: Vec<&str>, order_by: Vec<&str>, alias: Option<&str>, ) -> Self

WINDOW CLAUSE

pub fn add_column_with_cast( self, column: &str, new_alias: &str, data_type: &str, ) -> Self

CAST function

pub fn add_column_with_trim(self, column: &str, new_alias: &str) -> Self

TRIM funciton

pub fn add_column_with_regex( self, column: &str, pattern: &str, new_alias: &str, ) -> Self

REGEX function

pub fn display_query_plan(&self)

DISPLAY Query Plan

pub fn display_schema(&self)

Displays the current schema for debugging purposes.

pub fn display_query(&self)

Dipslaying query genereated from chained functions

pub async fn display(&self) -> Result<(), DataFusionError>

Display functions that display results to terminal

pub async fn write_to_parquet( &self, mode: &str, path: &str, options: Option<DataFrameWriteOptions>, ) -> ElusionResult<()>

Write the DataFrame to a Parquet file.

This function wraps DataFusion’s write_parquet method for easier usage.

Parameters

• mode: Specifies the write mode. Accepted values are:
  • "overwrite": Deletes existing files at the target path before writing.
  • "append": Appends to the existing Parquet file if it exists.
• path: The file path where the Parquet file will be saved.
• options: Optional write options for customizing the output.

Example

// Write to Parquet in overwrite mode
custom_df.write_to_parquet("overwrite", "output.parquet", None).await?;

// Write to Parquet in append mode
custom_df.write_to_parquet("append", "output.parquet", None).await?;

Errors

Returns a DataFusionError if the DataFrame execution or writing fails.

pub async fn write_to_csv( &self, mode: &str, path: &str, csv_options: CsvWriteOptions, ) -> ElusionResult<()>

Writes the DataFrame to a CSV file in either “overwrite” or “append” mode.

Arguments

• mode - The write mode, either “overwrite” or “append”.
• path - The file path where the CSV will be written.
• options - Optional DataFrameWriteOptions for customizing the write behavior.

Returns

• ElusionResult<()> - Ok(()) on success, or an ElusionError on failure.

Trait Implementations

impl Clone for CustomDataFrame

fn clone(&self) -> CustomDataFrame

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.